This project focuses on using an image dataset of eyes, which includes not only the iris but also the surrounding periocular region, to build Deep Convolutional Neural Network (DCNN) classifiers for identifying the sex of the individual in the image. A key aspect of the project is analyzing the image features extracted by the models to identify those most relevant for sex classification. Additionally, multiple DCNN architectures will be explored, with models trained on two versions of the same dataset: one with the original images and another with randomly augmented images. The training and performance of these models will be compared to evaluate the impact of data augmentation.
Identifying the sex of an individual from periocular images has important applications in various fields, from security and biometric authentication to medical diagnostics. In security, such models can enhance identity verification systems in situations where only partial or low-resolution facial data is available. Moreover, automated sex classification using periocular regions could improve the accuracy of systems used in demographics analysis and research. By building classifiers that focus on this subtle but ubiquitous piece of an individual's biometrics, this project addresses the need for more adaptable, hopefully privacy-conscious, and efficient systems in multiple real-world scenarios with limited or non-ideal data.
https://github.com/chill0121/Periocular_Sex_Classification
Note on the word-choice in this project:
The words "sex" and "gender" are often incorrectly used interchangeably in resources on this subject and broadly within society. Gender refers to characteristics largely assigned by cultural norms while sex are biological and physiologic characteristics. It has been decided that within this project the target class label used will be sex. However, depending on the efficacy of the model and image characteristics it uses to classify eyes, a strong argument could be made that the models are indeed classifying gender instead of sex, and I believe it is important to explore this distinction as it helps define what the model is actually identifying -- more on this in the Model Analysis and Feature Extraction (Section 8) and Conclusion (Section 10).
10.Conclusion
Source: https://www.kaggle.com/datasets/pavelbiz/eyes-rtte/data
Description:
By using OpenCV Haar Eye Cascade (https://github.com/anaustinbeing/haar-cascade-files/blob/master/haarcascade_eye.xml), by the Kaggle user pavelbiz, scraped eye images from the website https://ruskino.ru which is a IMDB-type website which houses actor, director, writer, etc information.
Contents:
Dataset size eventually parsed down to 5182 images at 56 x 56 x 3 pixels (2591 Female, 2591 Male). Reason and method is described in section 2.2.
import os
import sys
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from matplotlib.offsetbox import OffsetImage, AnnotationBbox
import matplotlib.patheffects as path_effects
from PIL import Image
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, ConfusionMatrixDisplay, auc, roc_curve
from sklearn.decomposition import PCA
import tensorflow as tf
from tf_explain.core.occlusion_sensitivity import OcclusionSensitivity
import visualkeras
print(f"Python version: {sys.version}")
packages = [pd, np, sns, sklearn, tf]
for package in packages:
print(f"{str(package).partition('from')[0]} using version: {package.__version__}")
Python version: 3.11.9 (main, Apr 2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)] <module 'pandas' using version: 2.1.4 <module 'numpy' using version: 1.26.4 <module 'seaborn' using version: 0.13.2 <module 'sklearn' using version: 1.3.2 <module 'tensorflow' using version: 2.16.2
Note: Two dataset characteristics were noticed during initial data import.
68 ... 71). This presents issues for splitting training and testing sets.The fixes chosen to overcome these two issues:
LANCZOS.def image_import(file_list, sex, image_size):
'''
Takes a list of image filenames and loads the images into a dictionary with the same filename(*).
(*) Adds up to 4 leading zeros to the name if they are not present.
Parameters:
file_list: List of filenames in the form ['name.extension', ...]
sex: String denoting the folder in ./data to find the images. ('Male' or 'Female')
image_size: Integer, will all images resize to this parameter.
All images are different sizes, median/mean of dataset is 56x56.
Returns:
eyes_dict: Dictionary whose keys are the filename with leading zeros and value is a numpy array of the image.
'''
eyes_dict = {}
# img_sizes = []
for file in file_list:
# Separate file extension and name.
number, _ = file.split('.')
number_new = number.zfill(4) # Add leading zeros.
# Only take odd numbers to remove more than 1 eye per person.
if int(number) % 2 == 1:
img = Image.open(f'./Data/{sex}/{number}.jpg')
# Used to check most common size.
#img_sizes.append(np.asarray(img).shape[0])
img = img.resize((image_size, image_size), Image.Resampling.LANCZOS)
eyes_dict[number_new] = np.asarray(img).flatten()
return eyes_dict #, image_sizes
img_size = 56 # Used throughout notebook.
female_files = os.listdir('./Data/Female')
male_files = os.listdir('./Data/male')
eyes_female = image_import(female_files, 'Female', image_size=img_size)
eyes_male = image_import(male_files, 'Male', image_size=img_size)
Let's take a look at a random image first and make sure it's the right size and expected format.
print(eyes_female['0011'].reshape(img_size,img_size,3).shape)
plt.imshow(Image.fromarray(eyes_female['0011'].reshape(img_size,img_size,3)))
plt.show()
(56, 56, 3)
Looks good - the shape of (56, 56, 3) indicates an 3-dimensional array representing RGB with no alpha channel.
Now, we should add the class labels to each image. This was made easier by keeping the two genders separate for now.
eyes_female_df = pd.DataFrame(eyes_female.items(), columns = ['eye_d', 'rgb_flat'])
eyes_male_df = pd.DataFrame(eyes_male.items(), columns = ['eye_d', 'rgb_flat'])
eyes_female_df['sex'] = 0
eyes_male_df['sex'] = 1
Now, we can see if the target variable is balanced or not.
print(f'Female Sample Size: {len(eyes_female_df)}')
print(f'Male Sample Size: {len(eyes_male_df)}')
Female Sample Size: 2591 Male Sample Size: 3173
This shows a fairly unbalanced dataset which is important to keep in mind for when we do the train / val / test set splits and model evaluations.
In fact, to simplify things, we can balance the dataset by removing the extra male images by randomly sampling the male eyes without replacement to match the number of female eyes. Then, the two dataframes will be combined into one.
# Balance classes, limiting the number of male eyes.
eyes_male_df = eyes_male_df.sample(n = len(eyes_female_df), random_state = 11, replace = False)
# Create dataframe with mixed sexes.
eyes_df = pd.concat([eyes_female_df, eyes_male_df]).reset_index(drop = True)
eyes_df
| eye_d | rgb_flat | sex | |
|---|---|---|---|
| 0 | 0823 | [216, 170, 136, 214, 168, 134, 212, 166, 133, ... | 0 |
| 1 | 4217 | [141, 77, 49, 140, 76, 48, 138, 77, 48, 137, 7... | 0 |
| 2 | 5109 | [244, 215, 199, 235, 206, 190, 165, 133, 118, ... | 0 |
| 3 | 4571 | [237, 210, 193, 234, 207, 190, 234, 205, 189, ... | 0 |
| 4 | 1409 | [175, 143, 130, 173, 141, 128, 172, 139, 130, ... | 0 |
| ... | ... | ... | ... |
| 5177 | 0579 | [151, 112, 104, 149, 110, 103, 153, 114, 109, ... | 1 |
| 5178 | 5191 | [210, 210, 210, 212, 212, 212, 209, 209, 209, ... | 1 |
| 5179 | 0425 | [163, 118, 87, 164, 119, 88, 170, 125, 94, 166... | 1 |
| 5180 | 5491 | [66, 44, 47, 48, 26, 30, 46, 25, 31, 44, 25, 3... | 1 |
| 5181 | 0821 | [221, 159, 138, 221, 158, 138, 217, 150, 130, ... | 1 |
5182 rows × 3 columns
Now eyes_df is the combined balanced Male and Female dataset.
It's important to normalize the image pixel intensities for use in the neural networks.
eyes_df['norm_flat'] = eyes_df['rgb_flat'] / 255
# Function to reshape arrays for processes / models that require image array.
def flat_to_array(flat_array):
'''
Takes a dataframe column (series) and reshapes flat array to image array (H x W x C).
'''
return flat_array.reshape(img_size,img_size,3)
To effectively train the models we need to create a validation set to evaluate training on, and of course we need a holdout dataset to do the final evaluations once all model tweaking has been finalized. The class balance will be maintained between all subsets by using the stratify parameter in train_test_split().
Split Ratios:
# Train and Test
X_train, X_test = train_test_split(eyes_df, test_size = 0.30, shuffle = True, stratify = eyes_df.sex, random_state = 11)
# Validation from Test
X_test, X_val = train_test_split(X_test, test_size = 0.60, shuffle = True, stratify = X_test.sex, random_state = 11)
# Create y.
y_train = X_train.sex
y_val = X_val.sex
y_test = X_test.sex
# Drop y from X.
X_train = X_train.drop(columns = ['sex'])
X_val = X_val.drop(columns = ['sex'])
X_test = X_test.drop(columns = ['sex'])
print(f'Train Set Size: {len(X_train)}')
print(f'Validation Set Size: {len(X_val)}')
print(f'Test Set Size: {len(X_test)}')
print('\n##################\n')
print('Target Class Balance:')
print('----Train:\n', y_train.value_counts())
print('----Validation:\n', y_val.value_counts())
print('----Test:\n', y_test.value_counts())
Train Set Size: 3627 Validation Set Size: 933 Test Set Size: 622 ################## Target Class Balance: ----Train: sex 0 1814 1 1813 Name: count, dtype: int64 ----Validation: sex 1 467 0 466 Name: count, dtype: int64 ----Test: sex 0 311 1 311 Name: count, dtype: int64
To avoid any data leakage, only images from the training set will be analyzed here.
# Colors for the target classes to stay consistent.
cmap_sex = {'Female' :'#439A86', 'Male': '#423e80'}
To get a better idea of the brightness/intensity of the images within the dataset we can take the mean pixel value across the whole image and plot that distribution.
# Find the mean pixel intensity of each image (down to one value along all axes).
mean_intensity = X_train.rgb_flat.apply(flat_to_array).apply(np.mean)
sns.histplot(mean_intensity, binwidth = 5, kde = True)
plt.xlabel('Mean Pixel Intensity Value')
plt.title('Pixel Intensity Histogram')
plt.show()
We can see that the distribution is centered around a pixel intensity of ~127 which means that the majority of images are not too light or dark. If we saw it positioned closer to the right, that means the photos would be darker and left, lighter.
We do however, see that some majority light and dark photos do exist with this dataset represented by the left and right tails respectively. We will keep this in mind when deciding which random image augmentations to apply during training.
Next, let's separate the color channels and plot their distributions.
mean_intensity_rgb = np.stack(X_train.rgb_flat.apply(flat_to_array).apply(np.mean, axis = 1).apply(np.mean, axis = 0).to_numpy())
# Plot the distribution of the mean intensities of individual RGB channels.
rgb_dict = {0:'Red', 1:'Green', 2:'Blue'}
for i in rgb_dict.keys():
sns.histplot(mean_intensity_rgb[:,i], color = rgb_dict[i], binwidth = 5, kde = True)#, ax = ax[i])
plt.title('Mean Pixel Intensity of Each Channel (RGB)')
plt.legend(list(rgb_dict.values()))
plt.show()
The red distribution being further right indicates that warmer-tones are more common in the dataset. This also might mean that the majority of the dataset is caucasian, which makes sense based on the data source.
It might be interesting to see these RGB channel distributions separated by target class (sex).
# Create a mask of the target class.
mask_sex = y_train == 0
# Plot the distribution of the mean intensities of individual channels in both Males and Females.
fig, ax = plt.subplots(1, 3, figsize = (14, 5), sharey = True)
for i in range(3):
# Female
sns.histplot(mean_intensity_rgb[:,i][mask_sex], kde = True, color = cmap_sex['Female'], ax = ax[i])
# Male
sns.histplot(mean_intensity_rgb[:,i][~mask_sex], kde = True, color = cmap_sex['Male'], ax = ax[i])
ax[i].set_title(f'{rgb_dict[i]}')
fig.supxlabel('Mean Pixel Intensity')
fig.suptitle('Color Channel Distributions by Class Label')
plt.legend(['Female', 'Male'])
plt.show()
There doesn't seem to be much difference in distributions between the two classes, but it's always important to check.
While we did import the images into memory before, utilizing a dataloader and applying random image augmentations at each batch call should increase performance of the models. These transformations will only be performed on the training set and will introduce much more diversity than the original dataset.
The following augmentations will be performed randomly:
BATCH_SIZE = 16
# Create a dataset with images and labels together.
# Train.
X_ds_train = tf.data.Dataset.from_tensor_slices(np.stack(X_train.norm_flat.values).reshape(len(X_train), img_size, img_size, 3))
y_ds_train = tf.data.Dataset.from_tensor_slices(y_train.values)
ds_train = tf.data.Dataset.zip((X_ds_train, y_ds_train))
# Val.
X_ds_val = tf.data.Dataset.from_tensor_slices(np.stack(X_val.norm_flat.values).reshape(len(X_val), img_size, img_size, 3))
y_ds_val = tf.data.Dataset.from_tensor_slices(y_val.values)
ds_val = tf.data.Dataset.zip((X_ds_val, y_ds_val))
# Transformation object.
data_augmentation = tf.keras.Sequential([
tf.keras.layers.RandomFlip('horizontal'),
tf.keras.layers.RandomRotation(0.15),
tf.keras.layers.RandomCrop(img_size, img_size),
tf.keras.layers.RandomBrightness(0.4, value_range = (0, 1)),
# Random saturation using lambda.
tf.keras.layers.Lambda(lambda x: tf.image.random_saturation(x, lower = 0, upper = 3))
])
# Autotune to set buffer size.
AUTOTUNE = tf.data.AUTOTUNE
# Function to setup the transformations within the dataloader.
def prepare(ds, shuffle = False, augment = False):
if shuffle:
ds = ds.shuffle(1000)
if augment:
ds = ds.map(lambda x, y: (data_augmentation(x, training = True), y), num_parallel_calls = AUTOTUNE)
# Prefetch buffer necessary here to ensure proper loading during training.
return ds.prefetch(buffer_size = AUTOTUNE)
# Training Set Final
ds_train_transformed = prepare(ds_train, shuffle = True, augment = True)
ds_train_transformed = ds_train_transformed.batch(BATCH_SIZE)
# Validation Set Final
ds_val = prepare(ds_val, shuffle = False, augment = False)
ds_val = ds_val.batch(BATCH_SIZE)
# For comparison to transformed training.
ds_train = prepare(ds_train, shuffle = True, augment = False)
ds_train = ds_train.batch(BATCH_SIZE)
for img, label in ds_train:
fig, ax = plt.subplots(2, 3, sharex = True, sharey = True)
img = (img.numpy() * 255).astype(np.uint8)
for i in range(6):
ax[i // 3, i % 3].imshow(img[i])
fig.suptitle('Non-Augmented Training Images')
break
for img, label in ds_train_transformed:
fig, ax = plt.subplots(2, 3, sharex = True, sharey = True)
img = (img.numpy() * 255).astype(np.uint8)
for i in range(6):
ax[i // 3, i % 3].imshow(img[i])
fig.suptitle('Augmented Training Images')
break
As was mentioned above, since the dataset has been class balanced, the metric we will evaluate all models is accuracy. Of course, the models will be trained using the training set and will utilize the validation set as the evaluation set during training. Validation set accuracy scores will be output after model training, and the final test set scores will be calculated in the Results section (9).
Also, each model will be trained on the training set WITH image augmentation, but also trained on the original dataset with the same dataloader settings, just without augmentations, so we can compare the performance of including these transformations at the end.
Here we will add a few functions to help visualize how the model perform during training.
def set_region_overlay(model_history_df, x_offset):
x_mid = ((model_history_df.index.stop-1) + model_history_df.val_loss.idxmin()) / 2
plt.text(x = x_mid - x_offset,
y = (plt.ylim()[0] + plt.ylim()[1]) / 2,
s = 'Early Stop',
rotation = 'horizontal',
weight = 'extra bold',
fontsize = 'large',
antialiased = True,
alpha = 1,
c = 'white',
bbox = dict(facecolor = 'black', edgecolor = 'black', boxstyle = 'round', alpha = 0.5))
return None
def plot_TF_training_history(model_history_df, plot_title = None):
# Find all epochs that callback ReduceLROnPlateau() occurred.
lr_change = model_history_df.learning_rate.shift(-1) != model_history_df.learning_rate
# Create color map and lines style map for train/val
plot_maps = {'cmap': {'accuracy': '#653096',
'loss': '#653096',
'val_accuracy': '#004a54',
'val_loss': '#004a54'},
'dashmap': {'accuracy': '',
'loss': (2,1),
'val_accuracy': '',
'val_loss': (2,1)}}
# Plot
fig, ax = plt.subplots(figsize = (10,6))
ax = sns.lineplot(model_history_df.drop(columns = ['learning_rate']).iloc[1:], palette = plot_maps['cmap'], dashes = plot_maps['dashmap'])
ax.set_xlabel('Epoch')
# Create secondary x-axis for Learning Rate changes.
sec_ax = ax.secondary_xaxis('top')
sec_ax.set_xticks(model_history_df[lr_change].index[:-1])
sec_ax.set_xticklabels([f'{x:.1e}' for x in model_history_df[lr_change].learning_rate[1:]])
sec_ax.tick_params(axis = 'x', which = 'major', labelsize = 7)
sec_ax.set_xlabel('Learning Rate Reductions')
# Create vertical line for each LR change.
for epoch in (model_history_df[lr_change].index[:-1]):
plt.axvline(x = epoch, c = '#d439ad', ls = (0, (5,5)))
# Create lines for best epoch/val_loss.
plt.axvline(x = (model_history_df.val_loss.idxmin()), c = '#f54260', ls = (0, (3,1,1,1)))
plt.axhline(y = (model_history_df.val_loss.min()), c = '#f54260', alpha = 0.3, ls = (0, (3,1,1,1)))
# Grey out epochs after early stop.
plt.axvspan(model_history_df.val_loss.idxmin(), model_history_df.index.stop-1, facecolor = 'black', alpha = 0.25)
plt.margins(x = 0)
set_region_overlay(model_history_df, 5)
if plot_title is not None:
fig.suptitle(plot_title)
plt.legend(loc = 'center')
plt.show()
return None
It's always important to formulate a baseline model to compare all models to.
Random Chance Baseline
Only two classes here, so the random chance baseline is 50%.
1/2
0.5
K-Nearest Neighbors (KNN) as Baseline
Since this dataset contains all eyes that are mostly centered in the same position, KNN is a quick and easy model to implement and compare more complex methods against.
# TODO: Implement KNN discriminant plot from sklearn.
mod_knn = KNeighborsClassifier(n_neighbors = 2, weights = 'distance', algorithm = 'brute', p = 2)
mod_knn.fit(np.stack(X_train.norm_flat), y_train)
y_pred_val_knn = mod_knn.predict(np.stack(X_val.norm_flat))
accuracy_score(y_val, y_pred_val_knn)
0.7738478027867095
77.38% accuracy on the validation set is pretty impressive for a model only using distance metrics on pixel intensities. This surprise performance is likely because most eyes in this dataset are in relatively the same position.
Now the prediction on the test set will be performed and saved for the results section after all tuning is completed.
# Predict on the test set.
y_pred_knn_proba = mod_knn.predict_proba(np.stack(X_test.norm_flat))
y_pred_knn = mod_knn.predict(np.stack(X_test.norm_flat))
Given that there are only two classes and one type of image (photos of eyes), it may be overlooked that a simple shallow dense network can potentially perform quite well on this classification task (see MNIST numbers dataset). While, it likely won't outperform a Convolutional Neural Network (CNN), it may be worth it to train what we'll call an FNN here to compare it to the CNN-based models.
We'll use two layers of dense nodes with ReLU activation functions and a light dropout (to mitigate overfitting) applied after activation of each of those two layers.
def build_fnn(input_shape = (img_size, img_size, 3)):
mod_fnn = tf.keras.Sequential([
tf.keras.Input(shape = input_shape, name = 'Imput Image'),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512),
tf.keras.layers.ReLU(),
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(128),
tf.keras.layers.ReLU(),
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(2, name = 'Predictions')])
return mod_fnn
mod_fnn = build_fnn()
mod_fnn.summary()
Model: "sequential_1"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ flatten (Flatten) │ (None, 9408) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 512) │ 4,817,408 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu (ReLU) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 128) │ 65,664 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_1 (ReLU) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ Predictions (Dense) │ (None, 2) │ 258 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 4,883,330 (18.63 MB)
Trainable params: 4,883,330 (18.63 MB)
Non-trainable params: 0 (0.00 B)
Model Specifications:
Below we can see the two "Callbacks" initialized. These are very important for how these models are being trained.
Details:
Both of these callbacks and all of their parameters were used and iteratively adjusted during training of all subsequent models.
loss_fx = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True) # Integer labels.
optimizer_param = 'adam'
val_freq = 1
n_epochs = 250
# Callbacks to use:
early_stop = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 25, verbose = 1, restore_best_weights = True)
reduce_lr_plateau = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss',
factor = 0.5,
patience = 5,
cooldown = 8,
min_lr = 0.0001,
verbose = 1)
mod_fnn.compile(optimizer = optimizer_param, loss = loss_fx, metrics = ['accuracy'])
mod_fnn_hist = mod_fnn.fit(
ds_train_transformed,
batch_size = BATCH_SIZE,
epochs = n_epochs,
verbose = "auto",
callbacks = [early_stop, reduce_lr_plateau],
#validation_split = 0.0,
validation_data = ds_val,
shuffle = True,
class_weight = None,
sample_weight = None,
initial_epoch = 0,
steps_per_epoch = None,
validation_steps = None,
validation_batch_size = None,
validation_freq = val_freq)
Epoch 1/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.5278 - loss: 2.1671 - val_accuracy: 0.5927 - val_loss: 0.6472 - learning_rate: 0.0010 Epoch 2/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 2s 11ms/step - accuracy: 0.6062 - loss: 0.6833 - val_accuracy: 0.6056 - val_loss: 0.6615 - learning_rate: 0.0010 Epoch 3/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.5961 - loss: 0.6758 - val_accuracy: 0.5756 - val_loss: 0.6795 - learning_rate: 0.0010 Epoch 4/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6171 - loss: 0.6577 - val_accuracy: 0.5005 - val_loss: 0.7131 - learning_rate: 0.0010 Epoch 5/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 2s 11ms/step - accuracy: 0.5448 - loss: 0.6892 - val_accuracy: 0.6742 - val_loss: 0.6361 - learning_rate: 0.0010 Epoch 6/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6135 - loss: 0.6547 - val_accuracy: 0.7085 - val_loss: 0.6061 - learning_rate: 0.0010 Epoch 7/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6528 - loss: 0.6352 - val_accuracy: 0.5273 - val_loss: 0.6927 - learning_rate: 0.0010 Epoch 8/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6448 - loss: 0.6366 - val_accuracy: 0.6956 - val_loss: 0.5839 - learning_rate: 0.0010 Epoch 9/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6817 - loss: 0.6157 - val_accuracy: 0.6506 - val_loss: 0.6462 - learning_rate: 0.0010 Epoch 10/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6851 - loss: 0.6095 - val_accuracy: 0.7556 - val_loss: 0.5567 - learning_rate: 0.0010 Epoch 11/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6576 - loss: 0.6245 - val_accuracy: 0.6699 - val_loss: 0.6260 - learning_rate: 0.0010 Epoch 12/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6724 - loss: 0.6188 - val_accuracy: 0.5348 - val_loss: 0.7265 - learning_rate: 0.0010 Epoch 13/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6414 - loss: 0.6337 - val_accuracy: 0.6924 - val_loss: 0.5991 - learning_rate: 0.0010 Epoch 14/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6734 - loss: 0.6304 - val_accuracy: 0.7192 - val_loss: 0.5567 - learning_rate: 0.0010 Epoch 15/250 226/227 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step - accuracy: 0.6744 - loss: 0.6125 Epoch 15: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257. 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6745 - loss: 0.6125 - val_accuracy: 0.7181 - val_loss: 0.5605 - learning_rate: 0.0010 Epoch 16/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6886 - loss: 0.5945 - val_accuracy: 0.7513 - val_loss: 0.5400 - learning_rate: 5.0000e-04 Epoch 17/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 2s 11ms/step - accuracy: 0.7160 - loss: 0.5832 - val_accuracy: 0.7556 - val_loss: 0.5297 - learning_rate: 5.0000e-04 Epoch 18/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7145 - loss: 0.5865 - val_accuracy: 0.7621 - val_loss: 0.5279 - learning_rate: 5.0000e-04 Epoch 19/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7130 - loss: 0.5730 - val_accuracy: 0.7074 - val_loss: 0.5564 - learning_rate: 5.0000e-04 Epoch 20/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7051 - loss: 0.5819 - val_accuracy: 0.7460 - val_loss: 0.5305 - learning_rate: 5.0000e-04 Epoch 21/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6952 - loss: 0.5907 - val_accuracy: 0.7556 - val_loss: 0.5296 - learning_rate: 5.0000e-04 Epoch 22/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7003 - loss: 0.5899 - val_accuracy: 0.6913 - val_loss: 0.5659 - learning_rate: 5.0000e-04 Epoch 23/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6935 - loss: 0.5938 - val_accuracy: 0.7331 - val_loss: 0.5399 - learning_rate: 5.0000e-04 Epoch 24/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7053 - loss: 0.5879 - val_accuracy: 0.7578 - val_loss: 0.5221 - learning_rate: 5.0000e-04 Epoch 25/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6995 - loss: 0.5798 - val_accuracy: 0.7556 - val_loss: 0.5289 - learning_rate: 5.0000e-04 Epoch 26/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6949 - loss: 0.5883 - val_accuracy: 0.6259 - val_loss: 0.6715 - learning_rate: 5.0000e-04 Epoch 27/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6971 - loss: 0.5866 - val_accuracy: 0.7567 - val_loss: 0.5245 - learning_rate: 5.0000e-04 Epoch 28/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7184 - loss: 0.5727 - val_accuracy: 0.7331 - val_loss: 0.5397 - learning_rate: 5.0000e-04 Epoch 29/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step - accuracy: 0.6896 - loss: 0.5951 Epoch 29: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628. 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.6897 - loss: 0.5950 - val_accuracy: 0.6860 - val_loss: 0.5609 - learning_rate: 5.0000e-04 Epoch 30/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7223 - loss: 0.5695 - val_accuracy: 0.7256 - val_loss: 0.5346 - learning_rate: 2.5000e-04 Epoch 31/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7203 - loss: 0.5684 - val_accuracy: 0.7588 - val_loss: 0.5087 - learning_rate: 2.5000e-04 Epoch 32/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7314 - loss: 0.5630 - val_accuracy: 0.7653 - val_loss: 0.5117 - learning_rate: 2.5000e-04 Epoch 33/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7172 - loss: 0.5780 - val_accuracy: 0.7642 - val_loss: 0.5322 - learning_rate: 2.5000e-04 Epoch 34/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7122 - loss: 0.5770 - val_accuracy: 0.7674 - val_loss: 0.5251 - learning_rate: 2.5000e-04 Epoch 35/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7180 - loss: 0.5665 - val_accuracy: 0.7642 - val_loss: 0.5087 - learning_rate: 2.5000e-04 Epoch 36/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7288 - loss: 0.5608 - val_accuracy: 0.7556 - val_loss: 0.5166 - learning_rate: 2.5000e-04 Epoch 37/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7148 - loss: 0.5716 - val_accuracy: 0.7578 - val_loss: 0.5150 - learning_rate: 2.5000e-04 Epoch 38/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7197 - loss: 0.5587 - val_accuracy: 0.7653 - val_loss: 0.5113 - learning_rate: 2.5000e-04 Epoch 39/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7154 - loss: 0.5623 - val_accuracy: 0.7203 - val_loss: 0.5411 - learning_rate: 2.5000e-04 Epoch 40/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7195 - loss: 0.5670 - val_accuracy: 0.7481 - val_loss: 0.5167 - learning_rate: 2.5000e-04 Epoch 41/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7432 - loss: 0.5531 - val_accuracy: 0.7760 - val_loss: 0.5075 - learning_rate: 2.5000e-04 Epoch 42/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7202 - loss: 0.5734 - val_accuracy: 0.7803 - val_loss: 0.5057 - learning_rate: 2.5000e-04 Epoch 43/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7083 - loss: 0.5624 - val_accuracy: 0.7610 - val_loss: 0.5035 - learning_rate: 2.5000e-04 Epoch 44/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7067 - loss: 0.5574 - val_accuracy: 0.7792 - val_loss: 0.4961 - learning_rate: 2.5000e-04 Epoch 45/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7352 - loss: 0.5504 - val_accuracy: 0.7546 - val_loss: 0.5094 - learning_rate: 2.5000e-04 Epoch 46/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7206 - loss: 0.5625 - val_accuracy: 0.7717 - val_loss: 0.4922 - learning_rate: 2.5000e-04 Epoch 47/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7306 - loss: 0.5493 - val_accuracy: 0.7878 - val_loss: 0.4931 - learning_rate: 2.5000e-04 Epoch 48/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7370 - loss: 0.5420 - val_accuracy: 0.7792 - val_loss: 0.5021 - learning_rate: 2.5000e-04 Epoch 49/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7432 - loss: 0.5403 - val_accuracy: 0.7728 - val_loss: 0.4857 - learning_rate: 2.5000e-04 Epoch 50/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7394 - loss: 0.5457 - val_accuracy: 0.7856 - val_loss: 0.5032 - learning_rate: 2.5000e-04 Epoch 51/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7357 - loss: 0.5541 - val_accuracy: 0.7749 - val_loss: 0.4863 - learning_rate: 2.5000e-04 Epoch 52/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7441 - loss: 0.5485 - val_accuracy: 0.7406 - val_loss: 0.5219 - learning_rate: 2.5000e-04 Epoch 53/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7263 - loss: 0.5487 - val_accuracy: 0.7867 - val_loss: 0.4823 - learning_rate: 2.5000e-04 Epoch 54/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7442 - loss: 0.5336 - val_accuracy: 0.7792 - val_loss: 0.4807 - learning_rate: 2.5000e-04 Epoch 55/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7309 - loss: 0.5418 - val_accuracy: 0.7567 - val_loss: 0.5111 - learning_rate: 2.5000e-04 Epoch 56/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7344 - loss: 0.5410 - val_accuracy: 0.7696 - val_loss: 0.5007 - learning_rate: 2.5000e-04 Epoch 57/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7358 - loss: 0.5387 - val_accuracy: 0.7867 - val_loss: 0.4766 - learning_rate: 2.5000e-04 Epoch 58/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7374 - loss: 0.5282 - val_accuracy: 0.7535 - val_loss: 0.5061 - learning_rate: 2.5000e-04 Epoch 59/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7340 - loss: 0.5403 - val_accuracy: 0.7803 - val_loss: 0.4748 - learning_rate: 2.5000e-04 Epoch 60/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7272 - loss: 0.5568 - val_accuracy: 0.7310 - val_loss: 0.5188 - learning_rate: 2.5000e-04 Epoch 61/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7365 - loss: 0.5432 - val_accuracy: 0.7460 - val_loss: 0.5156 - learning_rate: 2.5000e-04 Epoch 62/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7228 - loss: 0.5544 - val_accuracy: 0.7792 - val_loss: 0.4784 - learning_rate: 2.5000e-04 Epoch 63/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7327 - loss: 0.5305 - val_accuracy: 0.7706 - val_loss: 0.4859 - learning_rate: 2.5000e-04 Epoch 64/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step - accuracy: 0.7575 - loss: 0.5259 Epoch 64: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814. 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7575 - loss: 0.5259 - val_accuracy: 0.7856 - val_loss: 0.4804 - learning_rate: 2.5000e-04 Epoch 65/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7532 - loss: 0.5235 - val_accuracy: 0.7953 - val_loss: 0.4709 - learning_rate: 1.2500e-04 Epoch 66/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7425 - loss: 0.5269 - val_accuracy: 0.7867 - val_loss: 0.4709 - learning_rate: 1.2500e-04 Epoch 67/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7514 - loss: 0.5182 - val_accuracy: 0.7803 - val_loss: 0.4715 - learning_rate: 1.2500e-04 Epoch 68/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7430 - loss: 0.5236 - val_accuracy: 0.7942 - val_loss: 0.4691 - learning_rate: 1.2500e-04 Epoch 69/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7476 - loss: 0.5306 - val_accuracy: 0.7878 - val_loss: 0.4687 - learning_rate: 1.2500e-04 Epoch 70/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7573 - loss: 0.5227 - val_accuracy: 0.7942 - val_loss: 0.4684 - learning_rate: 1.2500e-04 Epoch 71/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7423 - loss: 0.5359 - val_accuracy: 0.7878 - val_loss: 0.4710 - learning_rate: 1.2500e-04 Epoch 72/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7535 - loss: 0.5218 - val_accuracy: 0.7974 - val_loss: 0.4651 - learning_rate: 1.2500e-04 Epoch 73/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7548 - loss: 0.5164 - val_accuracy: 0.7899 - val_loss: 0.4698 - learning_rate: 1.2500e-04 Epoch 74/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 2s 11ms/step - accuracy: 0.7643 - loss: 0.5066 - val_accuracy: 0.7856 - val_loss: 0.4674 - learning_rate: 1.2500e-04 Epoch 75/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7628 - loss: 0.4966 - val_accuracy: 0.7835 - val_loss: 0.4653 - learning_rate: 1.2500e-04 Epoch 76/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7610 - loss: 0.5117 - val_accuracy: 0.7835 - val_loss: 0.4660 - learning_rate: 1.2500e-04 Epoch 77/250 223/227 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step - accuracy: 0.7449 - loss: 0.5319 Epoch 77: ReduceLROnPlateau reducing learning rate to 0.0001. 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7450 - loss: 0.5317 - val_accuracy: 0.7996 - val_loss: 0.4651 - learning_rate: 1.2500e-04 Epoch 78/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7680 - loss: 0.5106 - val_accuracy: 0.7899 - val_loss: 0.4664 - learning_rate: 1.0000e-04 Epoch 79/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7484 - loss: 0.5119 - val_accuracy: 0.8017 - val_loss: 0.4626 - learning_rate: 1.0000e-04 Epoch 80/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7512 - loss: 0.5095 - val_accuracy: 0.7942 - val_loss: 0.4614 - learning_rate: 1.0000e-04 Epoch 81/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7394 - loss: 0.5417 - val_accuracy: 0.7556 - val_loss: 0.5083 - learning_rate: 1.0000e-04 Epoch 82/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7530 - loss: 0.5253 - val_accuracy: 0.7985 - val_loss: 0.4602 - learning_rate: 1.0000e-04 Epoch 83/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7509 - loss: 0.5223 - val_accuracy: 0.7878 - val_loss: 0.4686 - learning_rate: 1.0000e-04 Epoch 84/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7497 - loss: 0.5185 - val_accuracy: 0.7974 - val_loss: 0.4611 - learning_rate: 1.0000e-04 Epoch 85/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7741 - loss: 0.5092 - val_accuracy: 0.7996 - val_loss: 0.4605 - learning_rate: 1.0000e-04 Epoch 86/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7439 - loss: 0.5128 - val_accuracy: 0.8049 - val_loss: 0.4647 - learning_rate: 1.0000e-04 Epoch 87/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7612 - loss: 0.5120 - val_accuracy: 0.8049 - val_loss: 0.4596 - learning_rate: 1.0000e-04 Epoch 88/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7510 - loss: 0.5080 - val_accuracy: 0.8006 - val_loss: 0.4585 - learning_rate: 1.0000e-04 Epoch 89/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7573 - loss: 0.5123 - val_accuracy: 0.8060 - val_loss: 0.4556 - learning_rate: 1.0000e-04 Epoch 90/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7578 - loss: 0.5032 - val_accuracy: 0.7867 - val_loss: 0.4603 - learning_rate: 1.0000e-04 Epoch 91/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7661 - loss: 0.5007 - val_accuracy: 0.7867 - val_loss: 0.4624 - learning_rate: 1.0000e-04 Epoch 92/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7408 - loss: 0.5173 - val_accuracy: 0.7974 - val_loss: 0.4602 - learning_rate: 1.0000e-04 Epoch 93/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7578 - loss: 0.5027 - val_accuracy: 0.7771 - val_loss: 0.4704 - learning_rate: 1.0000e-04 Epoch 94/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7685 - loss: 0.4967 - val_accuracy: 0.7964 - val_loss: 0.4549 - learning_rate: 1.0000e-04 Epoch 95/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7637 - loss: 0.4983 - val_accuracy: 0.7867 - val_loss: 0.4587 - learning_rate: 1.0000e-04 Epoch 96/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7720 - loss: 0.4878 - val_accuracy: 0.7899 - val_loss: 0.4583 - learning_rate: 1.0000e-04 Epoch 97/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7608 - loss: 0.5024 - val_accuracy: 0.7996 - val_loss: 0.4570 - learning_rate: 1.0000e-04 Epoch 98/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7560 - loss: 0.5131 - val_accuracy: 0.8017 - val_loss: 0.4663 - learning_rate: 1.0000e-04 Epoch 99/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7682 - loss: 0.5004 - val_accuracy: 0.8006 - val_loss: 0.4563 - learning_rate: 1.0000e-04 Epoch 100/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7788 - loss: 0.4876 - val_accuracy: 0.7931 - val_loss: 0.4519 - learning_rate: 1.0000e-04 Epoch 101/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7668 - loss: 0.5059 - val_accuracy: 0.7996 - val_loss: 0.4624 - learning_rate: 1.0000e-04 Epoch 102/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7507 - loss: 0.5218 - val_accuracy: 0.7696 - val_loss: 0.4712 - learning_rate: 1.0000e-04 Epoch 103/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7647 - loss: 0.4958 - val_accuracy: 0.8049 - val_loss: 0.4533 - learning_rate: 1.0000e-04 Epoch 104/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7688 - loss: 0.4858 - val_accuracy: 0.8039 - val_loss: 0.4532 - learning_rate: 1.0000e-04 Epoch 105/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7608 - loss: 0.5013 - val_accuracy: 0.8039 - val_loss: 0.4507 - learning_rate: 1.0000e-04 Epoch 106/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7580 - loss: 0.4954 - val_accuracy: 0.7985 - val_loss: 0.4538 - learning_rate: 1.0000e-04 Epoch 107/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7628 - loss: 0.5039 - val_accuracy: 0.7996 - val_loss: 0.4555 - learning_rate: 1.0000e-04 Epoch 108/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7569 - loss: 0.5036 - val_accuracy: 0.8124 - val_loss: 0.4520 - learning_rate: 1.0000e-04 Epoch 109/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7563 - loss: 0.5058 - val_accuracy: 0.8071 - val_loss: 0.4521 - learning_rate: 1.0000e-04 Epoch 110/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7701 - loss: 0.5028 - val_accuracy: 0.8114 - val_loss: 0.4577 - learning_rate: 1.0000e-04 Epoch 111/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7793 - loss: 0.4833 - val_accuracy: 0.7996 - val_loss: 0.4529 - learning_rate: 1.0000e-04 Epoch 112/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7634 - loss: 0.5023 - val_accuracy: 0.8039 - val_loss: 0.4466 - learning_rate: 1.0000e-04 Epoch 113/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7796 - loss: 0.4849 - val_accuracy: 0.7910 - val_loss: 0.4517 - learning_rate: 1.0000e-04 Epoch 114/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7768 - loss: 0.4866 - val_accuracy: 0.8071 - val_loss: 0.4528 - learning_rate: 1.0000e-04 Epoch 115/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 2s 11ms/step - accuracy: 0.7723 - loss: 0.4868 - val_accuracy: 0.7910 - val_loss: 0.4551 - learning_rate: 1.0000e-04 Epoch 116/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 2s 11ms/step - accuracy: 0.7594 - loss: 0.4990 - val_accuracy: 0.8049 - val_loss: 0.4488 - learning_rate: 1.0000e-04 Epoch 117/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7585 - loss: 0.5068 - val_accuracy: 0.8071 - val_loss: 0.4499 - learning_rate: 1.0000e-04 Epoch 118/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7502 - loss: 0.4991 - val_accuracy: 0.8017 - val_loss: 0.4495 - learning_rate: 1.0000e-04 Epoch 119/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7747 - loss: 0.4887 - val_accuracy: 0.8081 - val_loss: 0.4473 - learning_rate: 1.0000e-04 Epoch 120/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7760 - loss: 0.4903 - val_accuracy: 0.8071 - val_loss: 0.4567 - learning_rate: 1.0000e-04 Epoch 121/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7530 - loss: 0.5066 - val_accuracy: 0.8017 - val_loss: 0.4453 - learning_rate: 1.0000e-04 Epoch 122/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7699 - loss: 0.4794 - val_accuracy: 0.7921 - val_loss: 0.4526 - learning_rate: 1.0000e-04 Epoch 123/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7645 - loss: 0.5037 - val_accuracy: 0.7856 - val_loss: 0.4560 - learning_rate: 1.0000e-04 Epoch 124/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7570 - loss: 0.4973 - val_accuracy: 0.7964 - val_loss: 0.4543 - learning_rate: 1.0000e-04 Epoch 125/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7652 - loss: 0.4843 - val_accuracy: 0.7867 - val_loss: 0.4642 - learning_rate: 1.0000e-04 Epoch 126/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7527 - loss: 0.4993 - val_accuracy: 0.7964 - val_loss: 0.4457 - learning_rate: 1.0000e-04 Epoch 127/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7753 - loss: 0.4791 - val_accuracy: 0.8049 - val_loss: 0.4412 - learning_rate: 1.0000e-04 Epoch 128/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7745 - loss: 0.4897 - val_accuracy: 0.8028 - val_loss: 0.4493 - learning_rate: 1.0000e-04 Epoch 129/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7479 - loss: 0.5164 - val_accuracy: 0.8028 - val_loss: 0.4448 - learning_rate: 1.0000e-04 Epoch 130/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7654 - loss: 0.4893 - val_accuracy: 0.8146 - val_loss: 0.4498 - learning_rate: 1.0000e-04 Epoch 131/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7679 - loss: 0.4846 - val_accuracy: 0.8039 - val_loss: 0.4575 - learning_rate: 1.0000e-04 Epoch 132/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7634 - loss: 0.5007 - val_accuracy: 0.7910 - val_loss: 0.4551 - learning_rate: 1.0000e-04 Epoch 133/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7667 - loss: 0.5045 - val_accuracy: 0.8017 - val_loss: 0.4552 - learning_rate: 1.0000e-04 Epoch 134/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7610 - loss: 0.4902 - val_accuracy: 0.7728 - val_loss: 0.4647 - learning_rate: 1.0000e-04 Epoch 135/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7531 - loss: 0.5082 - val_accuracy: 0.7760 - val_loss: 0.4710 - learning_rate: 1.0000e-04 Epoch 136/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7501 - loss: 0.5070 - val_accuracy: 0.8103 - val_loss: 0.4427 - learning_rate: 1.0000e-04 Epoch 137/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7751 - loss: 0.4925 - val_accuracy: 0.8039 - val_loss: 0.4390 - learning_rate: 1.0000e-04 Epoch 138/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7701 - loss: 0.4965 - val_accuracy: 0.8017 - val_loss: 0.4426 - learning_rate: 1.0000e-04 Epoch 139/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7551 - loss: 0.5084 - val_accuracy: 0.8049 - val_loss: 0.4449 - learning_rate: 1.0000e-04 Epoch 140/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7639 - loss: 0.4881 - val_accuracy: 0.8135 - val_loss: 0.4471 - learning_rate: 1.0000e-04 Epoch 141/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7726 - loss: 0.4913 - val_accuracy: 0.8081 - val_loss: 0.4419 - learning_rate: 1.0000e-04 Epoch 142/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7661 - loss: 0.4876 - val_accuracy: 0.8049 - val_loss: 0.4503 - learning_rate: 1.0000e-04 Epoch 143/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7761 - loss: 0.4749 - val_accuracy: 0.8103 - val_loss: 0.4504 - learning_rate: 1.0000e-04 Epoch 144/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7705 - loss: 0.4923 - val_accuracy: 0.7942 - val_loss: 0.4531 - learning_rate: 1.0000e-04 Epoch 145/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7738 - loss: 0.4890 - val_accuracy: 0.7921 - val_loss: 0.4541 - learning_rate: 1.0000e-04 Epoch 146/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7696 - loss: 0.4947 - val_accuracy: 0.8028 - val_loss: 0.4487 - learning_rate: 1.0000e-04 Epoch 147/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7552 - loss: 0.4883 - val_accuracy: 0.7803 - val_loss: 0.4604 - learning_rate: 1.0000e-04 Epoch 148/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7682 - loss: 0.4809 - val_accuracy: 0.8156 - val_loss: 0.4465 - learning_rate: 1.0000e-04 Epoch 149/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7665 - loss: 0.4955 - val_accuracy: 0.8081 - val_loss: 0.4468 - learning_rate: 1.0000e-04 Epoch 150/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7814 - loss: 0.4832 - val_accuracy: 0.7964 - val_loss: 0.4431 - learning_rate: 1.0000e-04 Epoch 151/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7702 - loss: 0.4834 - val_accuracy: 0.8006 - val_loss: 0.4473 - learning_rate: 1.0000e-04 Epoch 152/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7740 - loss: 0.4856 - val_accuracy: 0.8006 - val_loss: 0.4520 - learning_rate: 1.0000e-04 Epoch 153/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7644 - loss: 0.4965 - val_accuracy: 0.7846 - val_loss: 0.4498 - learning_rate: 1.0000e-04 Epoch 154/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7857 - loss: 0.4678 - val_accuracy: 0.7599 - val_loss: 0.4881 - learning_rate: 1.0000e-04 Epoch 155/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7743 - loss: 0.4826 - val_accuracy: 0.7942 - val_loss: 0.4487 - learning_rate: 1.0000e-04 Epoch 156/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7676 - loss: 0.4826 - val_accuracy: 0.7899 - val_loss: 0.4555 - learning_rate: 1.0000e-04 Epoch 157/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7713 - loss: 0.4790 - val_accuracy: 0.8081 - val_loss: 0.4387 - learning_rate: 1.0000e-04 Epoch 158/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7774 - loss: 0.4782 - val_accuracy: 0.8124 - val_loss: 0.4413 - learning_rate: 1.0000e-04 Epoch 159/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7708 - loss: 0.4990 - val_accuracy: 0.8028 - val_loss: 0.4385 - learning_rate: 1.0000e-04 Epoch 160/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7810 - loss: 0.4860 - val_accuracy: 0.8114 - val_loss: 0.4378 - learning_rate: 1.0000e-04 Epoch 161/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7745 - loss: 0.4835 - val_accuracy: 0.8124 - val_loss: 0.4430 - learning_rate: 1.0000e-04 Epoch 162/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7818 - loss: 0.4753 - val_accuracy: 0.7856 - val_loss: 0.4503 - learning_rate: 1.0000e-04 Epoch 163/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7724 - loss: 0.4810 - val_accuracy: 0.7985 - val_loss: 0.4499 - learning_rate: 1.0000e-04 Epoch 164/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7654 - loss: 0.4956 - val_accuracy: 0.7953 - val_loss: 0.4535 - learning_rate: 1.0000e-04 Epoch 165/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7550 - loss: 0.5046 - val_accuracy: 0.8081 - val_loss: 0.4365 - learning_rate: 1.0000e-04 Epoch 166/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7741 - loss: 0.4819 - val_accuracy: 0.7942 - val_loss: 0.4449 - learning_rate: 1.0000e-04 Epoch 167/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7684 - loss: 0.4816 - val_accuracy: 0.8006 - val_loss: 0.4454 - learning_rate: 1.0000e-04 Epoch 168/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7774 - loss: 0.4784 - val_accuracy: 0.8146 - val_loss: 0.4245 - learning_rate: 1.0000e-04 Epoch 169/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7840 - loss: 0.4832 - val_accuracy: 0.8114 - val_loss: 0.4391 - learning_rate: 1.0000e-04 Epoch 170/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7658 - loss: 0.4833 - val_accuracy: 0.8103 - val_loss: 0.4282 - learning_rate: 1.0000e-04 Epoch 171/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7759 - loss: 0.4665 - val_accuracy: 0.7631 - val_loss: 0.4703 - learning_rate: 1.0000e-04 Epoch 172/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7722 - loss: 0.4823 - val_accuracy: 0.8049 - val_loss: 0.4369 - learning_rate: 1.0000e-04 Epoch 173/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7795 - loss: 0.4775 - val_accuracy: 0.8114 - val_loss: 0.4217 - learning_rate: 1.0000e-04 Epoch 174/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7839 - loss: 0.4551 - val_accuracy: 0.8167 - val_loss: 0.4239 - learning_rate: 1.0000e-04 Epoch 175/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7782 - loss: 0.4653 - val_accuracy: 0.8274 - val_loss: 0.4318 - learning_rate: 1.0000e-04 Epoch 176/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7856 - loss: 0.4577 - val_accuracy: 0.8135 - val_loss: 0.4335 - learning_rate: 1.0000e-04 Epoch 177/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7794 - loss: 0.4797 - val_accuracy: 0.8189 - val_loss: 0.4188 - learning_rate: 1.0000e-04 Epoch 178/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7800 - loss: 0.4512 - val_accuracy: 0.8156 - val_loss: 0.4285 - learning_rate: 1.0000e-04 Epoch 179/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7743 - loss: 0.4778 - val_accuracy: 0.8156 - val_loss: 0.4330 - learning_rate: 1.0000e-04 Epoch 180/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7794 - loss: 0.4738 - val_accuracy: 0.8199 - val_loss: 0.4272 - learning_rate: 1.0000e-04 Epoch 181/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7875 - loss: 0.4631 - val_accuracy: 0.8199 - val_loss: 0.4263 - learning_rate: 1.0000e-04 Epoch 182/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7817 - loss: 0.4611 - val_accuracy: 0.8039 - val_loss: 0.4243 - learning_rate: 1.0000e-04 Epoch 183/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7861 - loss: 0.4508 - val_accuracy: 0.8135 - val_loss: 0.4153 - learning_rate: 1.0000e-04 Epoch 184/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7828 - loss: 0.4670 - val_accuracy: 0.7964 - val_loss: 0.4351 - learning_rate: 1.0000e-04 Epoch 185/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7883 - loss: 0.4573 - val_accuracy: 0.8242 - val_loss: 0.4157 - learning_rate: 1.0000e-04 Epoch 186/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7772 - loss: 0.4777 - val_accuracy: 0.8060 - val_loss: 0.4279 - learning_rate: 1.0000e-04 Epoch 187/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7961 - loss: 0.4574 - val_accuracy: 0.8081 - val_loss: 0.4321 - learning_rate: 1.0000e-04 Epoch 188/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7880 - loss: 0.4511 - val_accuracy: 0.8264 - val_loss: 0.4218 - learning_rate: 1.0000e-04 Epoch 189/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7847 - loss: 0.4639 - val_accuracy: 0.7953 - val_loss: 0.4353 - learning_rate: 1.0000e-04 Epoch 190/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7891 - loss: 0.4468 - val_accuracy: 0.8135 - val_loss: 0.4242 - learning_rate: 1.0000e-04 Epoch 191/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7921 - loss: 0.4514 - val_accuracy: 0.8156 - val_loss: 0.4064 - learning_rate: 1.0000e-04 Epoch 192/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8040 - loss: 0.4481 - val_accuracy: 0.8178 - val_loss: 0.4227 - learning_rate: 1.0000e-04 Epoch 193/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7813 - loss: 0.4816 - val_accuracy: 0.8285 - val_loss: 0.4154 - learning_rate: 1.0000e-04 Epoch 194/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7892 - loss: 0.4651 - val_accuracy: 0.8167 - val_loss: 0.4100 - learning_rate: 1.0000e-04 Epoch 195/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7811 - loss: 0.4677 - val_accuracy: 0.8071 - val_loss: 0.4214 - learning_rate: 1.0000e-04 Epoch 196/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7883 - loss: 0.4530 - val_accuracy: 0.8167 - val_loss: 0.4233 - learning_rate: 1.0000e-04 Epoch 197/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7895 - loss: 0.4555 - val_accuracy: 0.8232 - val_loss: 0.4130 - learning_rate: 1.0000e-04 Epoch 198/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7910 - loss: 0.4449 - val_accuracy: 0.8253 - val_loss: 0.4177 - learning_rate: 1.0000e-04 Epoch 199/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7835 - loss: 0.4625 - val_accuracy: 0.7931 - val_loss: 0.4277 - learning_rate: 1.0000e-04 Epoch 200/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8050 - loss: 0.4465 - val_accuracy: 0.8264 - val_loss: 0.4070 - learning_rate: 1.0000e-04 Epoch 201/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7956 - loss: 0.4425 - val_accuracy: 0.7856 - val_loss: 0.4433 - learning_rate: 1.0000e-04 Epoch 202/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7791 - loss: 0.4516 - val_accuracy: 0.8307 - val_loss: 0.4095 - learning_rate: 1.0000e-04 Epoch 203/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8006 - loss: 0.4374 - val_accuracy: 0.8264 - val_loss: 0.4145 - learning_rate: 1.0000e-04 Epoch 204/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8048 - loss: 0.4436 - val_accuracy: 0.8178 - val_loss: 0.4074 - learning_rate: 1.0000e-04 Epoch 205/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7994 - loss: 0.4363 - val_accuracy: 0.7814 - val_loss: 0.4597 - learning_rate: 1.0000e-04 Epoch 206/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7955 - loss: 0.4491 - val_accuracy: 0.8178 - val_loss: 0.4078 - learning_rate: 1.0000e-04 Epoch 207/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7895 - loss: 0.4371 - val_accuracy: 0.8242 - val_loss: 0.4150 - learning_rate: 1.0000e-04 Epoch 208/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7959 - loss: 0.4583 - val_accuracy: 0.8307 - val_loss: 0.4008 - learning_rate: 1.0000e-04 Epoch 209/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8027 - loss: 0.4349 - val_accuracy: 0.8317 - val_loss: 0.4048 - learning_rate: 1.0000e-04 Epoch 210/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7983 - loss: 0.4416 - val_accuracy: 0.8103 - val_loss: 0.4164 - learning_rate: 1.0000e-04 Epoch 211/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8099 - loss: 0.4303 - val_accuracy: 0.8371 - val_loss: 0.4056 - learning_rate: 1.0000e-04 Epoch 212/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8087 - loss: 0.4316 - val_accuracy: 0.8092 - val_loss: 0.4151 - learning_rate: 1.0000e-04 Epoch 213/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7921 - loss: 0.4533 - val_accuracy: 0.8317 - val_loss: 0.4019 - learning_rate: 1.0000e-04 Epoch 214/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8043 - loss: 0.4350 - val_accuracy: 0.8167 - val_loss: 0.4077 - learning_rate: 1.0000e-04 Epoch 215/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7855 - loss: 0.4604 - val_accuracy: 0.8285 - val_loss: 0.4015 - learning_rate: 1.0000e-04 Epoch 216/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8062 - loss: 0.4304 - val_accuracy: 0.8285 - val_loss: 0.4011 - learning_rate: 1.0000e-04 Epoch 217/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7957 - loss: 0.4414 - val_accuracy: 0.8360 - val_loss: 0.4039 - learning_rate: 1.0000e-04 Epoch 218/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7975 - loss: 0.4349 - val_accuracy: 0.8328 - val_loss: 0.4013 - learning_rate: 1.0000e-04 Epoch 219/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8064 - loss: 0.4379 - val_accuracy: 0.8028 - val_loss: 0.4181 - learning_rate: 1.0000e-04 Epoch 220/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7870 - loss: 0.4493 - val_accuracy: 0.8307 - val_loss: 0.4015 - learning_rate: 1.0000e-04 Epoch 221/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7920 - loss: 0.4479 - val_accuracy: 0.8360 - val_loss: 0.3958 - learning_rate: 1.0000e-04 Epoch 222/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7856 - loss: 0.4563 - val_accuracy: 0.8242 - val_loss: 0.4042 - learning_rate: 1.0000e-04 Epoch 223/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8082 - loss: 0.4232 - val_accuracy: 0.8253 - val_loss: 0.3953 - learning_rate: 1.0000e-04 Epoch 224/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8033 - loss: 0.4452 - val_accuracy: 0.8264 - val_loss: 0.3991 - learning_rate: 1.0000e-04 Epoch 225/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8128 - loss: 0.4171 - val_accuracy: 0.8242 - val_loss: 0.4054 - learning_rate: 1.0000e-04 Epoch 226/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7957 - loss: 0.4422 - val_accuracy: 0.8221 - val_loss: 0.4003 - learning_rate: 1.0000e-04 Epoch 227/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8090 - loss: 0.4264 - val_accuracy: 0.8339 - val_loss: 0.3975 - learning_rate: 1.0000e-04 Epoch 228/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 12ms/step - accuracy: 0.8071 - loss: 0.4329 - val_accuracy: 0.8221 - val_loss: 0.3977 - learning_rate: 1.0000e-04 Epoch 229/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8003 - loss: 0.4319 - val_accuracy: 0.8274 - val_loss: 0.3898 - learning_rate: 1.0000e-04 Epoch 230/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8015 - loss: 0.4346 - val_accuracy: 0.8146 - val_loss: 0.4085 - learning_rate: 1.0000e-04 Epoch 231/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7905 - loss: 0.4344 - val_accuracy: 0.8371 - val_loss: 0.3951 - learning_rate: 1.0000e-04 Epoch 232/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7985 - loss: 0.4396 - val_accuracy: 0.8328 - val_loss: 0.4040 - learning_rate: 1.0000e-04 Epoch 233/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8099 - loss: 0.4217 - val_accuracy: 0.8382 - val_loss: 0.3894 - learning_rate: 1.0000e-04 Epoch 234/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7971 - loss: 0.4401 - val_accuracy: 0.8392 - val_loss: 0.3939 - learning_rate: 1.0000e-04 Epoch 235/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8054 - loss: 0.4271 - val_accuracy: 0.8071 - val_loss: 0.4178 - learning_rate: 1.0000e-04 Epoch 236/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8055 - loss: 0.4347 - val_accuracy: 0.8264 - val_loss: 0.4077 - learning_rate: 1.0000e-04 Epoch 237/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8120 - loss: 0.4236 - val_accuracy: 0.8349 - val_loss: 0.3888 - learning_rate: 1.0000e-04 Epoch 238/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8086 - loss: 0.4239 - val_accuracy: 0.8349 - val_loss: 0.3958 - learning_rate: 1.0000e-04 Epoch 239/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7989 - loss: 0.4348 - val_accuracy: 0.8392 - val_loss: 0.3925 - learning_rate: 1.0000e-04 Epoch 240/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8070 - loss: 0.4302 - val_accuracy: 0.8264 - val_loss: 0.3987 - learning_rate: 1.0000e-04 Epoch 241/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8044 - loss: 0.4312 - val_accuracy: 0.8435 - val_loss: 0.3870 - learning_rate: 1.0000e-04 Epoch 242/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8169 - loss: 0.4115 - val_accuracy: 0.8339 - val_loss: 0.3970 - learning_rate: 1.0000e-04 Epoch 243/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8017 - loss: 0.4287 - val_accuracy: 0.8274 - val_loss: 0.3913 - learning_rate: 1.0000e-04 Epoch 244/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.7910 - loss: 0.4426 - val_accuracy: 0.8446 - val_loss: 0.3875 - learning_rate: 1.0000e-04 Epoch 245/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8055 - loss: 0.4329 - val_accuracy: 0.7996 - val_loss: 0.4276 - learning_rate: 1.0000e-04 Epoch 246/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8146 - loss: 0.4261 - val_accuracy: 0.8403 - val_loss: 0.3880 - learning_rate: 1.0000e-04 Epoch 247/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 12ms/step - accuracy: 0.7991 - loss: 0.4301 - val_accuracy: 0.8274 - val_loss: 0.3910 - learning_rate: 1.0000e-04 Epoch 248/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8087 - loss: 0.4301 - val_accuracy: 0.8360 - val_loss: 0.3883 - learning_rate: 1.0000e-04 Epoch 249/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - accuracy: 0.8126 - loss: 0.4226 - val_accuracy: 0.8210 - val_loss: 0.4035 - learning_rate: 1.0000e-04 Epoch 250/250 227/227 ━━━━━━━━━━━━━━━━━━━━ 3s 12ms/step - accuracy: 0.8124 - loss: 0.4296 - val_accuracy: 0.8403 - val_loss: 0.3922 - learning_rate: 1.0000e-04 Restoring model weights from the end of the best epoch: 241.
# Create dataframe of model fit history.
mod_fnn_hist_df = pd.DataFrame().from_dict(mod_fnn_hist.history, orient = 'columns')
plot_TF_training_history(mod_fnn_hist_df, 'KNN - Training History')
This looks great, and the model didn't overfit to the training data. The model seems slightly unstable, but that is to be expected especially with the random augmentations being applied to the training data at each epoch because this type of model is not invariant to the types of augmentations being performed.
Looking at the training history here, the model could likely be trainied for a few more epochs, but through tuning it I know that this is about as stable as it gets and has essentially converged here so we will move on.
y_pred_val_fnn_proba = mod_fnn.predict(ds_val, verbose = "auto", callbacks = None)
y_pred_val_fnn = y_pred_val_fnn_proba.argmax(axis = 1)
accuracy_score(y_val, y_pred_val_fnn)
59/59 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step
0.8435155412647374
84.35% accuracy on the validation set is a great score for image classification using only dense layers!
Now the prediction on the test set will be performed and saved for the results section after all tuning is completed.
y_pred_fnn_proba = mod_fnn.predict(np.stack(X_test.norm_flat.apply(flat_to_array)), verbose = "auto", callbacks = None)
y_pred_fnn = y_pred_fnn_proba.argmax(axis = 1)
20/20 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Train the same model but use the training set that is not augmented/transformed for comparison at the end. Muted outputs for notebook clarity.
mod_fnn_no_aug = build_fnn()
mod_fnn_no_aug.compile(loss = loss_fx,
optimizer = optimizer_param,
metrics = ['accuracy'])
# Callbacks to use (same as above):
early_stop = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 25, verbose = 1, restore_best_weights = True)
reduce_lr_plateau = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, patience = 5, cooldown = 8, min_lr = 0.0001, verbose = 0)
mod_fnn_no_aug_hist = mod_fnn_no_aug.fit(ds_train,
batch_size = BATCH_SIZE,
epochs = n_epochs,
verbose = 0,
callbacks = [early_stop, reduce_lr_plateau],
validation_split = 0.0,
validation_data = ds_val,
shuffle = True,
class_weight = None,
sample_weight = None,
initial_epoch = 0,
steps_per_epoch = None,
validation_steps = None,
validation_batch_size = None,
validation_freq = val_freq)
y_pred_val_fnn_no_aug_proba = mod_fnn_no_aug.predict(ds_val, verbose = 0, callbacks = None)
y_pred_val_fnn_no_aug = y_pred_val_fnn_no_aug_proba.argmax(axis = 1)
y_pred_fnn_no_aug_proba = mod_fnn_no_aug.predict(np.stack(X_test.norm_flat.apply(flat_to_array)), verbose = 0, callbacks = None)
y_pred_fnn_no_aug = y_pred_fnn_no_aug_proba.argmax(axis = 1)
Epoch 100: early stopping Restoring model weights from the end of the best epoch: 75.
We can see training without the variance that randomly augmenting introduces into the dataset, heavily reduces the training time. This model reverted back to epoch 75 while the previous model, with image augmentations, trained until 241. We'll see how this effects performance in the results section.
Model Specifications:
The model below showcases the benefits of stacking multiple 3x3 convolutional layers instead of utilizing larger kernel sizes.
Consider a network using the same number of filters with larger kernel sizes (such as 32 filters of 5x5 and 96 filters of 7x7) would require 20,473,314 trainable parameters. This model, by distributing the same number of filters across multiple layers of 3x3 kernels, achieves the same receptive field with only 7,501,106 parameters. This significant reduction is due to the fact that larger kernels introduces redundant parameter calculations without adding unique value to the model's learning capability.
An additional advantage of using multiple 3x3 kernels is the introduction of activation functions between layers, which increases the model’s non-linearity and allows it to capture more complex patterns in the data compared to larger single-layer kernels.
These factors allow the use of much deeper networks while maintaining computational efficiency, which is why modern architectures predominantly use stacked 3x3 kernels.
def build_cnn(input_shape = (img_size, img_size, 3)):
model = tf.keras.models.Sequential([
tf.keras.Input(shape = input_shape, name = 'Image Array'),
# First block (2).
tf.keras.layers.Conv2D(16, kernel_size = (3,3), padding = 'same'),
tf.keras.layers.ReLU(),
tf.keras.layers.Conv2D(16, kernel_size = (3,3), padding = 'same'),
tf.keras.layers.ReLU(),
tf.keras.layers.MaxPooling2D((2,2), strides = 2),
# Second block (3)
tf.keras.layers.Conv2D(32, kernel_size = (3,3), padding = 'same'),
tf.keras.layers.ReLU(),
tf.keras.layers.Conv2D(32, kernel_size = (3,3), padding = 'same'),
tf.keras.layers.ReLU(),
tf.keras.layers.Conv2D(32, kernel_size = (3,3), padding = 'same'),
tf.keras.layers.ReLU(),
tf.keras.layers.MaxPooling2D((2,2), strides = 2),
# Classifier block (2).
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1024),
tf.keras.layers.ReLU(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1024),
tf.keras.layers.ReLU(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(2, name = 'Predictions')])
return model
mod_cnn = build_cnn()
mod_cnn.summary()
Model: "sequential_3"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv2d (Conv2D) │ (None, 56, 56, 16) │ 448 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_4 (ReLU) │ (None, 56, 56, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_1 (Conv2D) │ (None, 56, 56, 16) │ 2,320 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_5 (ReLU) │ (None, 56, 56, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d (MaxPooling2D) │ (None, 28, 28, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_2 (Conv2D) │ (None, 28, 28, 32) │ 4,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_6 (ReLU) │ (None, 28, 28, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_3 (Conv2D) │ (None, 28, 28, 32) │ 9,248 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_7 (ReLU) │ (None, 28, 28, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_4 (Conv2D) │ (None, 28, 28, 32) │ 9,248 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_8 (ReLU) │ (None, 28, 28, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_1 (MaxPooling2D) │ (None, 14, 14, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten_2 (Flatten) │ (None, 6272) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_4 (Dense) │ (None, 1024) │ 6,423,552 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_9 (ReLU) │ (None, 1024) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_4 (Dropout) │ (None, 1024) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_5 (Dense) │ (None, 1024) │ 1,049,600 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ re_lu_10 (ReLU) │ (None, 1024) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_5 (Dropout) │ (None, 1024) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ Predictions (Dense) │ (None, 2) │ 2,050 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 7,501,106 (28.61 MB)
Trainable params: 7,501,106 (28.61 MB)
Non-trainable params: 0 (0.00 B)
visualkeras.layered_view(mod_cnn, legend = True)
loss_fx = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True) # Integer labels.
optimizer_param = 'sgd'
val_freq = 1
n_epochs = 300
# Callbacks to use:
early_stop = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 40, verbose = 1, restore_best_weights = True)
reduce_lr_plateau = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss',
factor = 0.5,
patience = 5,
cooldown = 8,
min_lr = 0.0001,
verbose = 1)
mod_cnn.compile(loss = loss_fx,
optimizer = optimizer_param,
metrics = ['accuracy'])
mod_cnn_hist = mod_cnn.fit(ds_train_transformed,
batch_size = BATCH_SIZE,
epochs = n_epochs,
verbose = "auto",
callbacks = [early_stop, reduce_lr_plateau],
validation_split = 0.0,
validation_data = ds_val,
shuffle = True,
class_weight = None,
sample_weight = None,
initial_epoch = 0,
steps_per_epoch = None,
validation_steps = None,
validation_batch_size = None,
validation_freq = val_freq)
Epoch 1/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.5011 - loss: 0.6938 - val_accuracy: 0.5102 - val_loss: 0.6913 - learning_rate: 0.0100 Epoch 2/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.5154 - loss: 0.6925 - val_accuracy: 0.5852 - val_loss: 0.6893 - learning_rate: 0.0100 Epoch 3/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.5191 - loss: 0.6922 - val_accuracy: 0.6120 - val_loss: 0.6847 - learning_rate: 0.0100 Epoch 4/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.5469 - loss: 0.6889 - val_accuracy: 0.6452 - val_loss: 0.6814 - learning_rate: 0.0100 Epoch 5/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.5495 - loss: 0.6881 - val_accuracy: 0.6710 - val_loss: 0.6748 - learning_rate: 0.0100 Epoch 6/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.5820 - loss: 0.6793 - val_accuracy: 0.7063 - val_loss: 0.6572 - learning_rate: 0.0100 Epoch 7/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.6005 - loss: 0.6722 - val_accuracy: 0.7631 - val_loss: 0.6198 - learning_rate: 0.0100 Epoch 8/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.6253 - loss: 0.6511 - val_accuracy: 0.7149 - val_loss: 0.5777 - learning_rate: 0.0100 Epoch 9/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.6540 - loss: 0.6289 - val_accuracy: 0.7621 - val_loss: 0.5655 - learning_rate: 0.0100 Epoch 10/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.6820 - loss: 0.5970 - val_accuracy: 0.7567 - val_loss: 0.5013 - learning_rate: 0.0100 Epoch 11/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.6841 - loss: 0.5850 - val_accuracy: 0.7760 - val_loss: 0.4885 - learning_rate: 0.0100 Epoch 12/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.6954 - loss: 0.5744 - val_accuracy: 0.7792 - val_loss: 0.4834 - learning_rate: 0.0100 Epoch 13/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.7263 - loss: 0.5557 - val_accuracy: 0.7985 - val_loss: 0.4540 - learning_rate: 0.0100 Epoch 14/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.7243 - loss: 0.5405 - val_accuracy: 0.7942 - val_loss: 0.4566 - learning_rate: 0.0100 Epoch 15/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.7226 - loss: 0.5424 - val_accuracy: 0.7320 - val_loss: 0.5362 - learning_rate: 0.0100 Epoch 16/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.7399 - loss: 0.5198 - val_accuracy: 0.7567 - val_loss: 0.4724 - learning_rate: 0.0100 Epoch 17/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.7350 - loss: 0.5234 - val_accuracy: 0.8135 - val_loss: 0.4254 - learning_rate: 0.0100 Epoch 18/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.7470 - loss: 0.5144 - val_accuracy: 0.7921 - val_loss: 0.4480 - learning_rate: 0.0100 Epoch 19/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.7591 - loss: 0.5013 - val_accuracy: 0.8253 - val_loss: 0.4139 - learning_rate: 0.0100 Epoch 20/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.7626 - loss: 0.4899 - val_accuracy: 0.8274 - val_loss: 0.4054 - learning_rate: 0.0100 Epoch 21/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.7649 - loss: 0.4918 - val_accuracy: 0.8071 - val_loss: 0.4250 - learning_rate: 0.0100 Epoch 22/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.7655 - loss: 0.4863 - val_accuracy: 0.8167 - val_loss: 0.4035 - learning_rate: 0.0100 Epoch 23/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.7694 - loss: 0.4739 - val_accuracy: 0.8478 - val_loss: 0.3720 - learning_rate: 0.0100 Epoch 24/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.7823 - loss: 0.4741 - val_accuracy: 0.8414 - val_loss: 0.3547 - learning_rate: 0.0100 Epoch 25/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.7871 - loss: 0.4499 - val_accuracy: 0.8639 - val_loss: 0.3458 - learning_rate: 0.0100 Epoch 26/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.7864 - loss: 0.4616 - val_accuracy: 0.8542 - val_loss: 0.3329 - learning_rate: 0.0100 Epoch 27/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.7856 - loss: 0.4494 - val_accuracy: 0.8628 - val_loss: 0.3377 - learning_rate: 0.0100 Epoch 28/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.7922 - loss: 0.4409 - val_accuracy: 0.8542 - val_loss: 0.3508 - learning_rate: 0.0100 Epoch 29/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8033 - loss: 0.4319 - val_accuracy: 0.8457 - val_loss: 0.3438 - learning_rate: 0.0100 Epoch 30/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.7900 - loss: 0.4338 - val_accuracy: 0.8789 - val_loss: 0.3046 - learning_rate: 0.0100 Epoch 31/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8133 - loss: 0.4117 - val_accuracy: 0.8725 - val_loss: 0.2968 - learning_rate: 0.0100 Epoch 32/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.7944 - loss: 0.4390 - val_accuracy: 0.8757 - val_loss: 0.2964 - learning_rate: 0.0100 Epoch 33/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8291 - loss: 0.4056 - val_accuracy: 0.8424 - val_loss: 0.3535 - learning_rate: 0.0100 Epoch 34/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8314 - loss: 0.3855 - val_accuracy: 0.8778 - val_loss: 0.2943 - learning_rate: 0.0100 Epoch 35/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.8138 - loss: 0.4047 - val_accuracy: 0.8950 - val_loss: 0.2706 - learning_rate: 0.0100 Epoch 36/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8231 - loss: 0.3887 - val_accuracy: 0.8917 - val_loss: 0.2744 - learning_rate: 0.0100 Epoch 37/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8256 - loss: 0.3958 - val_accuracy: 0.8778 - val_loss: 0.2995 - learning_rate: 0.0100 Epoch 38/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8426 - loss: 0.3588 - val_accuracy: 0.8917 - val_loss: 0.2440 - learning_rate: 0.0100 Epoch 39/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8346 - loss: 0.3701 - val_accuracy: 0.8907 - val_loss: 0.2743 - learning_rate: 0.0100 Epoch 40/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8373 - loss: 0.3659 - val_accuracy: 0.9164 - val_loss: 0.2297 - learning_rate: 0.0100 Epoch 41/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8382 - loss: 0.3633 - val_accuracy: 0.9025 - val_loss: 0.2460 - learning_rate: 0.0100 Epoch 42/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8512 - loss: 0.3483 - val_accuracy: 0.9100 - val_loss: 0.2303 - learning_rate: 0.0100 Epoch 43/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8413 - loss: 0.3498 - val_accuracy: 0.8585 - val_loss: 0.3174 - learning_rate: 0.0100 Epoch 44/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8450 - loss: 0.3560 - val_accuracy: 0.9046 - val_loss: 0.2404 - learning_rate: 0.0100 Epoch 45/300 226/227 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step - accuracy: 0.8476 - loss: 0.3401 Epoch 45: ReduceLROnPlateau reducing learning rate to 0.004999999888241291. 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8476 - loss: 0.3401 - val_accuracy: 0.9014 - val_loss: 0.2374 - learning_rate: 0.0100 Epoch 46/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8725 - loss: 0.3162 - val_accuracy: 0.9132 - val_loss: 0.2332 - learning_rate: 0.0050 Epoch 47/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8632 - loss: 0.3204 - val_accuracy: 0.9218 - val_loss: 0.2122 - learning_rate: 0.0050 Epoch 48/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8713 - loss: 0.2996 - val_accuracy: 0.9078 - val_loss: 0.2127 - learning_rate: 0.0050 Epoch 49/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8519 - loss: 0.3181 - val_accuracy: 0.9121 - val_loss: 0.2198 - learning_rate: 0.0050 Epoch 50/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8679 - loss: 0.3034 - val_accuracy: 0.9143 - val_loss: 0.2086 - learning_rate: 0.0050 Epoch 51/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8655 - loss: 0.3207 - val_accuracy: 0.9175 - val_loss: 0.2065 - learning_rate: 0.0050 Epoch 52/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.8789 - loss: 0.2945 - val_accuracy: 0.8746 - val_loss: 0.2998 - learning_rate: 0.0050 Epoch 53/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.8767 - loss: 0.2996 - val_accuracy: 0.9196 - val_loss: 0.1997 - learning_rate: 0.0050 Epoch 54/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8786 - loss: 0.2854 - val_accuracy: 0.9250 - val_loss: 0.1888 - learning_rate: 0.0050 Epoch 55/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.8765 - loss: 0.2987 - val_accuracy: 0.9196 - val_loss: 0.1927 - learning_rate: 0.0050 Epoch 56/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8754 - loss: 0.2901 - val_accuracy: 0.9132 - val_loss: 0.2104 - learning_rate: 0.0050 Epoch 57/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8884 - loss: 0.2794 - val_accuracy: 0.9185 - val_loss: 0.1992 - learning_rate: 0.0050 Epoch 58/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8749 - loss: 0.2927 - val_accuracy: 0.9175 - val_loss: 0.1999 - learning_rate: 0.0050 Epoch 59/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8772 - loss: 0.2900 - val_accuracy: 0.9239 - val_loss: 0.1839 - learning_rate: 0.0050 Epoch 60/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8814 - loss: 0.2912 - val_accuracy: 0.9089 - val_loss: 0.2196 - learning_rate: 0.0050 Epoch 61/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8890 - loss: 0.2832 - val_accuracy: 0.9185 - val_loss: 0.1918 - learning_rate: 0.0050 Epoch 62/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8780 - loss: 0.2926 - val_accuracy: 0.9282 - val_loss: 0.1931 - learning_rate: 0.0050 Epoch 63/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8823 - loss: 0.2797 - val_accuracy: 0.9143 - val_loss: 0.2004 - learning_rate: 0.0050 Epoch 64/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8863 - loss: 0.2766 - val_accuracy: 0.9196 - val_loss: 0.1817 - learning_rate: 0.0050 Epoch 65/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.8914 - loss: 0.2683 - val_accuracy: 0.9185 - val_loss: 0.1980 - learning_rate: 0.0050 Epoch 66/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.8794 - loss: 0.2799 - val_accuracy: 0.9153 - val_loss: 0.1942 - learning_rate: 0.0050 Epoch 67/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.8887 - loss: 0.2684 - val_accuracy: 0.9250 - val_loss: 0.1917 - learning_rate: 0.0050 Epoch 68/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.8917 - loss: 0.2583 - val_accuracy: 0.9239 - val_loss: 0.1793 - learning_rate: 0.0050 Epoch 69/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9003 - loss: 0.2638 - val_accuracy: 0.9282 - val_loss: 0.1779 - learning_rate: 0.0050 Epoch 70/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8830 - loss: 0.2740 - val_accuracy: 0.9293 - val_loss: 0.1807 - learning_rate: 0.0050 Epoch 71/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8905 - loss: 0.2580 - val_accuracy: 0.9357 - val_loss: 0.1728 - learning_rate: 0.0050 Epoch 72/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8900 - loss: 0.2610 - val_accuracy: 0.9078 - val_loss: 0.2174 - learning_rate: 0.0050 Epoch 73/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8875 - loss: 0.2769 - val_accuracy: 0.9335 - val_loss: 0.1798 - learning_rate: 0.0050 Epoch 74/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.8993 - loss: 0.2427 - val_accuracy: 0.9293 - val_loss: 0.1737 - learning_rate: 0.0050 Epoch 75/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8943 - loss: 0.2575 - val_accuracy: 0.9293 - val_loss: 0.1723 - learning_rate: 0.0050 Epoch 76/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8883 - loss: 0.2710 - val_accuracy: 0.9314 - val_loss: 0.1756 - learning_rate: 0.0050 Epoch 77/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8879 - loss: 0.2689 - val_accuracy: 0.9143 - val_loss: 0.1914 - learning_rate: 0.0050 Epoch 78/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.8976 - loss: 0.2505 - val_accuracy: 0.9271 - val_loss: 0.1732 - learning_rate: 0.0050 Epoch 79/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8868 - loss: 0.2580 - val_accuracy: 0.9207 - val_loss: 0.1936 - learning_rate: 0.0050 Epoch 80/300 226/227 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step - accuracy: 0.9006 - loss: 0.2337 Epoch 80: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455. 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9006 - loss: 0.2338 - val_accuracy: 0.9335 - val_loss: 0.1901 - learning_rate: 0.0050 Epoch 81/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9058 - loss: 0.2443 - val_accuracy: 0.9400 - val_loss: 0.1680 - learning_rate: 0.0025 Epoch 82/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9059 - loss: 0.2497 - val_accuracy: 0.9271 - val_loss: 0.1787 - learning_rate: 0.0025 Epoch 83/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.8978 - loss: 0.2499 - val_accuracy: 0.9357 - val_loss: 0.1680 - learning_rate: 0.0025 Epoch 84/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9144 - loss: 0.2263 - val_accuracy: 0.9207 - val_loss: 0.1822 - learning_rate: 0.0025 Epoch 85/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9060 - loss: 0.2320 - val_accuracy: 0.9335 - val_loss: 0.1759 - learning_rate: 0.0025 Epoch 86/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9049 - loss: 0.2328 - val_accuracy: 0.9378 - val_loss: 0.1650 - learning_rate: 0.0025 Epoch 87/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9051 - loss: 0.2347 - val_accuracy: 0.9293 - val_loss: 0.1752 - learning_rate: 0.0025 Epoch 88/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9081 - loss: 0.2220 - val_accuracy: 0.9303 - val_loss: 0.1744 - learning_rate: 0.0025 Epoch 89/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.8998 - loss: 0.2385 - val_accuracy: 0.9325 - val_loss: 0.1779 - learning_rate: 0.0025 Epoch 90/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9048 - loss: 0.2296 - val_accuracy: 0.9250 - val_loss: 0.1777 - learning_rate: 0.0025 Epoch 91/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9115 - loss: 0.2244 - val_accuracy: 0.9303 - val_loss: 0.1718 - learning_rate: 0.0025 Epoch 92/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step - accuracy: 0.8981 - loss: 0.2377 Epoch 92: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228. 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.8981 - loss: 0.2377 - val_accuracy: 0.9228 - val_loss: 0.1749 - learning_rate: 0.0025 Epoch 93/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9005 - loss: 0.2319 - val_accuracy: 0.9293 - val_loss: 0.1833 - learning_rate: 0.0012 Epoch 94/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9185 - loss: 0.2157 - val_accuracy: 0.9325 - val_loss: 0.1639 - learning_rate: 0.0012 Epoch 95/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9100 - loss: 0.2236 - val_accuracy: 0.9325 - val_loss: 0.1610 - learning_rate: 0.0012 Epoch 96/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9136 - loss: 0.2208 - val_accuracy: 0.9314 - val_loss: 0.1637 - learning_rate: 0.0012 Epoch 97/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9165 - loss: 0.2130 - val_accuracy: 0.9335 - val_loss: 0.1872 - learning_rate: 0.0012 Epoch 98/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9192 - loss: 0.2042 - val_accuracy: 0.9368 - val_loss: 0.1552 - learning_rate: 0.0012 Epoch 99/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9078 - loss: 0.2229 - val_accuracy: 0.9357 - val_loss: 0.1645 - learning_rate: 0.0012 Epoch 100/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9058 - loss: 0.2288 - val_accuracy: 0.9346 - val_loss: 0.1648 - learning_rate: 0.0012 Epoch 101/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.9139 - loss: 0.2095 - val_accuracy: 0.9303 - val_loss: 0.1745 - learning_rate: 0.0012 Epoch 102/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9172 - loss: 0.2083 - val_accuracy: 0.9357 - val_loss: 0.1602 - learning_rate: 0.0012 Epoch 103/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9076 - loss: 0.2213 - val_accuracy: 0.9346 - val_loss: 0.1700 - learning_rate: 0.0012 Epoch 104/300 226/227 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step - accuracy: 0.9139 - loss: 0.2243 Epoch 104: ReduceLROnPlateau reducing learning rate to 0.0006249999860301614. 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9139 - loss: 0.2241 - val_accuracy: 0.9432 - val_loss: 0.1580 - learning_rate: 0.0012 Epoch 105/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9044 - loss: 0.2167 - val_accuracy: 0.9368 - val_loss: 0.1582 - learning_rate: 6.2500e-04 Epoch 106/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9060 - loss: 0.2190 - val_accuracy: 0.9346 - val_loss: 0.1588 - learning_rate: 6.2500e-04 Epoch 107/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9257 - loss: 0.1957 - val_accuracy: 0.9411 - val_loss: 0.1572 - learning_rate: 6.2500e-04 Epoch 108/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9143 - loss: 0.2089 - val_accuracy: 0.9378 - val_loss: 0.1607 - learning_rate: 6.2500e-04 Epoch 109/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9148 - loss: 0.2041 - val_accuracy: 0.9400 - val_loss: 0.1572 - learning_rate: 6.2500e-04 Epoch 110/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9247 - loss: 0.1999 - val_accuracy: 0.9432 - val_loss: 0.1548 - learning_rate: 6.2500e-04 Epoch 111/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9148 - loss: 0.2049 - val_accuracy: 0.9411 - val_loss: 0.1559 - learning_rate: 6.2500e-04 Epoch 112/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9098 - loss: 0.2224 - val_accuracy: 0.9400 - val_loss: 0.1604 - learning_rate: 6.2500e-04 Epoch 113/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9176 - loss: 0.2087 - val_accuracy: 0.9443 - val_loss: 0.1574 - learning_rate: 6.2500e-04 Epoch 114/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9219 - loss: 0.2075 - val_accuracy: 0.9411 - val_loss: 0.1580 - learning_rate: 6.2500e-04 Epoch 115/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9095 - loss: 0.2170 - val_accuracy: 0.9443 - val_loss: 0.1591 - learning_rate: 6.2500e-04 Epoch 116/300 226/227 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step - accuracy: 0.9250 - loss: 0.1910 Epoch 116: ReduceLROnPlateau reducing learning rate to 0.0003124999930150807. 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9250 - loss: 0.1910 - val_accuracy: 0.9443 - val_loss: 0.1607 - learning_rate: 6.2500e-04 Epoch 117/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9159 - loss: 0.2129 - val_accuracy: 0.9421 - val_loss: 0.1563 - learning_rate: 3.1250e-04 Epoch 118/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9225 - loss: 0.1998 - val_accuracy: 0.9411 - val_loss: 0.1603 - learning_rate: 3.1250e-04 Epoch 119/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9168 - loss: 0.2067 - val_accuracy: 0.9368 - val_loss: 0.1586 - learning_rate: 3.1250e-04 Epoch 120/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9029 - loss: 0.2201 - val_accuracy: 0.9357 - val_loss: 0.1620 - learning_rate: 3.1250e-04 Epoch 121/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9237 - loss: 0.1910 - val_accuracy: 0.9400 - val_loss: 0.1601 - learning_rate: 3.1250e-04 Epoch 122/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9252 - loss: 0.2046 - val_accuracy: 0.9389 - val_loss: 0.1582 - learning_rate: 3.1250e-04 Epoch 123/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9134 - loss: 0.2142 - val_accuracy: 0.9411 - val_loss: 0.1568 - learning_rate: 3.1250e-04 Epoch 124/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9092 - loss: 0.2057 - val_accuracy: 0.9389 - val_loss: 0.1554 - learning_rate: 3.1250e-04 Epoch 125/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9237 - loss: 0.1954 - val_accuracy: 0.9432 - val_loss: 0.1567 - learning_rate: 3.1250e-04 Epoch 126/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9279 - loss: 0.1928 - val_accuracy: 0.9421 - val_loss: 0.1550 - learning_rate: 3.1250e-04 Epoch 127/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9282 - loss: 0.1977 - val_accuracy: 0.9411 - val_loss: 0.1560 - learning_rate: 3.1250e-04 Epoch 128/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step - accuracy: 0.9109 - loss: 0.2052 Epoch 128: ReduceLROnPlateau reducing learning rate to 0.00015624999650754035. 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9109 - loss: 0.2051 - val_accuracy: 0.9421 - val_loss: 0.1583 - learning_rate: 3.1250e-04 Epoch 129/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9149 - loss: 0.2078 - val_accuracy: 0.9421 - val_loss: 0.1593 - learning_rate: 1.5625e-04 Epoch 130/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9188 - loss: 0.1984 - val_accuracy: 0.9368 - val_loss: 0.1595 - learning_rate: 1.5625e-04 Epoch 131/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9161 - loss: 0.1998 - val_accuracy: 0.9421 - val_loss: 0.1576 - learning_rate: 1.5625e-04 Epoch 132/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9159 - loss: 0.2001 - val_accuracy: 0.9453 - val_loss: 0.1575 - learning_rate: 1.5625e-04 Epoch 133/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9134 - loss: 0.2151 - val_accuracy: 0.9421 - val_loss: 0.1587 - learning_rate: 1.5625e-04 Epoch 134/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9028 - loss: 0.2401 - val_accuracy: 0.9432 - val_loss: 0.1575 - learning_rate: 1.5625e-04 Epoch 135/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9223 - loss: 0.1980 - val_accuracy: 0.9443 - val_loss: 0.1578 - learning_rate: 1.5625e-04 Epoch 136/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9255 - loss: 0.1923 - val_accuracy: 0.9443 - val_loss: 0.1565 - learning_rate: 1.5625e-04 Epoch 137/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9278 - loss: 0.1983 - val_accuracy: 0.9432 - val_loss: 0.1573 - learning_rate: 1.5625e-04 Epoch 138/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9265 - loss: 0.1852 - val_accuracy: 0.9400 - val_loss: 0.1593 - learning_rate: 1.5625e-04 Epoch 139/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9230 - loss: 0.1943 - val_accuracy: 0.9453 - val_loss: 0.1550 - learning_rate: 1.5625e-04 Epoch 140/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 28ms/step - accuracy: 0.9138 - loss: 0.2101 Epoch 140: ReduceLROnPlateau reducing learning rate to 0.0001. 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.9139 - loss: 0.2100 - val_accuracy: 0.9443 - val_loss: 0.1551 - learning_rate: 1.5625e-04 Epoch 141/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 31ms/step - accuracy: 0.9158 - loss: 0.1997 - val_accuracy: 0.9453 - val_loss: 0.1559 - learning_rate: 1.0000e-04 Epoch 142/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9273 - loss: 0.1824 - val_accuracy: 0.9432 - val_loss: 0.1573 - learning_rate: 1.0000e-04 Epoch 143/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9241 - loss: 0.1929 - val_accuracy: 0.9432 - val_loss: 0.1561 - learning_rate: 1.0000e-04 Epoch 144/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9091 - loss: 0.2150 - val_accuracy: 0.9421 - val_loss: 0.1557 - learning_rate: 1.0000e-04 Epoch 145/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9184 - loss: 0.1891 - val_accuracy: 0.9453 - val_loss: 0.1558 - learning_rate: 1.0000e-04 Epoch 146/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9138 - loss: 0.2102 - val_accuracy: 0.9443 - val_loss: 0.1562 - learning_rate: 1.0000e-04 Epoch 147/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9170 - loss: 0.1964 - val_accuracy: 0.9443 - val_loss: 0.1559 - learning_rate: 1.0000e-04 Epoch 148/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9155 - loss: 0.2053 - val_accuracy: 0.9443 - val_loss: 0.1548 - learning_rate: 1.0000e-04 Epoch 149/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9191 - loss: 0.1953 - val_accuracy: 0.9443 - val_loss: 0.1565 - learning_rate: 1.0000e-04 Epoch 150/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9145 - loss: 0.2035 - val_accuracy: 0.9421 - val_loss: 0.1567 - learning_rate: 1.0000e-04 Epoch 151/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9251 - loss: 0.1833 - val_accuracy: 0.9411 - val_loss: 0.1566 - learning_rate: 1.0000e-04 Epoch 152/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9224 - loss: 0.1890 - val_accuracy: 0.9443 - val_loss: 0.1565 - learning_rate: 1.0000e-04 Epoch 153/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9220 - loss: 0.1931 - val_accuracy: 0.9432 - val_loss: 0.1565 - learning_rate: 1.0000e-04 Epoch 154/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9175 - loss: 0.2009 - val_accuracy: 0.9432 - val_loss: 0.1568 - learning_rate: 1.0000e-04 Epoch 155/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9164 - loss: 0.1907 - val_accuracy: 0.9464 - val_loss: 0.1552 - learning_rate: 1.0000e-04 Epoch 156/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9248 - loss: 0.1993 - val_accuracy: 0.9432 - val_loss: 0.1568 - learning_rate: 1.0000e-04 Epoch 157/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9198 - loss: 0.1976 - val_accuracy: 0.9432 - val_loss: 0.1573 - learning_rate: 1.0000e-04 Epoch 158/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9185 - loss: 0.1963 - val_accuracy: 0.9411 - val_loss: 0.1566 - learning_rate: 1.0000e-04 Epoch 159/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9220 - loss: 0.1838 - val_accuracy: 0.9464 - val_loss: 0.1559 - learning_rate: 1.0000e-04 Epoch 160/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9201 - loss: 0.1907 - val_accuracy: 0.9475 - val_loss: 0.1557 - learning_rate: 1.0000e-04 Epoch 161/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9262 - loss: 0.2019 - val_accuracy: 0.9443 - val_loss: 0.1554 - learning_rate: 1.0000e-04 Epoch 162/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9149 - loss: 0.2030 - val_accuracy: 0.9464 - val_loss: 0.1547 - learning_rate: 1.0000e-04 Epoch 163/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9130 - loss: 0.1967 - val_accuracy: 0.9453 - val_loss: 0.1556 - learning_rate: 1.0000e-04 Epoch 164/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9204 - loss: 0.1982 - val_accuracy: 0.9453 - val_loss: 0.1555 - learning_rate: 1.0000e-04 Epoch 165/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9286 - loss: 0.1903 - val_accuracy: 0.9443 - val_loss: 0.1550 - learning_rate: 1.0000e-04 Epoch 166/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9183 - loss: 0.1950 - val_accuracy: 0.9453 - val_loss: 0.1555 - learning_rate: 1.0000e-04 Epoch 167/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9303 - loss: 0.1849 - val_accuracy: 0.9464 - val_loss: 0.1547 - learning_rate: 1.0000e-04 Epoch 168/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9240 - loss: 0.2035 - val_accuracy: 0.9432 - val_loss: 0.1562 - learning_rate: 1.0000e-04 Epoch 169/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9099 - loss: 0.1958 - val_accuracy: 0.9421 - val_loss: 0.1550 - learning_rate: 1.0000e-04 Epoch 170/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9170 - loss: 0.2066 - val_accuracy: 0.9432 - val_loss: 0.1562 - learning_rate: 1.0000e-04 Epoch 171/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9183 - loss: 0.2022 - val_accuracy: 0.9443 - val_loss: 0.1555 - learning_rate: 1.0000e-04 Epoch 172/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9218 - loss: 0.1989 - val_accuracy: 0.9411 - val_loss: 0.1559 - learning_rate: 1.0000e-04 Epoch 173/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9183 - loss: 0.1931 - val_accuracy: 0.9443 - val_loss: 0.1560 - learning_rate: 1.0000e-04 Epoch 174/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9179 - loss: 0.2098 - val_accuracy: 0.9432 - val_loss: 0.1556 - learning_rate: 1.0000e-04 Epoch 175/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9236 - loss: 0.1935 - val_accuracy: 0.9443 - val_loss: 0.1554 - learning_rate: 1.0000e-04 Epoch 176/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9256 - loss: 0.1927 - val_accuracy: 0.9443 - val_loss: 0.1554 - learning_rate: 1.0000e-04 Epoch 177/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9195 - loss: 0.1993 - val_accuracy: 0.9443 - val_loss: 0.1560 - learning_rate: 1.0000e-04 Epoch 178/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9179 - loss: 0.1950 - val_accuracy: 0.9443 - val_loss: 0.1570 - learning_rate: 1.0000e-04 Epoch 179/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9221 - loss: 0.2092 - val_accuracy: 0.9443 - val_loss: 0.1568 - learning_rate: 1.0000e-04 Epoch 180/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9144 - loss: 0.2207 - val_accuracy: 0.9443 - val_loss: 0.1564 - learning_rate: 1.0000e-04 Epoch 181/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9124 - loss: 0.2072 - val_accuracy: 0.9432 - val_loss: 0.1557 - learning_rate: 1.0000e-04 Epoch 182/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9248 - loss: 0.1844 - val_accuracy: 0.9411 - val_loss: 0.1550 - learning_rate: 1.0000e-04 Epoch 183/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9193 - loss: 0.1933 - val_accuracy: 0.9453 - val_loss: 0.1554 - learning_rate: 1.0000e-04 Epoch 184/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9207 - loss: 0.1991 - val_accuracy: 0.9453 - val_loss: 0.1553 - learning_rate: 1.0000e-04 Epoch 185/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9228 - loss: 0.1920 - val_accuracy: 0.9443 - val_loss: 0.1570 - learning_rate: 1.0000e-04 Epoch 186/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9199 - loss: 0.1998 - val_accuracy: 0.9453 - val_loss: 0.1560 - learning_rate: 1.0000e-04 Epoch 187/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9183 - loss: 0.1993 - val_accuracy: 0.9443 - val_loss: 0.1558 - learning_rate: 1.0000e-04 Epoch 188/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9192 - loss: 0.2025 - val_accuracy: 0.9464 - val_loss: 0.1547 - learning_rate: 1.0000e-04 Epoch 189/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9210 - loss: 0.1974 - val_accuracy: 0.9464 - val_loss: 0.1553 - learning_rate: 1.0000e-04 Epoch 190/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9156 - loss: 0.2052 - val_accuracy: 0.9464 - val_loss: 0.1552 - learning_rate: 1.0000e-04 Epoch 191/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9194 - loss: 0.1851 - val_accuracy: 0.9389 - val_loss: 0.1552 - learning_rate: 1.0000e-04 Epoch 192/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9177 - loss: 0.1998 - val_accuracy: 0.9453 - val_loss: 0.1548 - learning_rate: 1.0000e-04 Epoch 193/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9187 - loss: 0.1984 - val_accuracy: 0.9432 - val_loss: 0.1547 - learning_rate: 1.0000e-04 Epoch 194/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9247 - loss: 0.1857 - val_accuracy: 0.9443 - val_loss: 0.1559 - learning_rate: 1.0000e-04 Epoch 195/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9200 - loss: 0.2033 - val_accuracy: 0.9443 - val_loss: 0.1557 - learning_rate: 1.0000e-04 Epoch 196/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9240 - loss: 0.1857 - val_accuracy: 0.9421 - val_loss: 0.1540 - learning_rate: 1.0000e-04 Epoch 197/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9235 - loss: 0.1902 - val_accuracy: 0.9432 - val_loss: 0.1544 - learning_rate: 1.0000e-04 Epoch 198/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9126 - loss: 0.1977 - val_accuracy: 0.9443 - val_loss: 0.1547 - learning_rate: 1.0000e-04 Epoch 199/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9152 - loss: 0.2214 - val_accuracy: 0.9421 - val_loss: 0.1539 - learning_rate: 1.0000e-04 Epoch 200/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9353 - loss: 0.1871 - val_accuracy: 0.9443 - val_loss: 0.1540 - learning_rate: 1.0000e-04 Epoch 201/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9179 - loss: 0.1936 - val_accuracy: 0.9443 - val_loss: 0.1535 - learning_rate: 1.0000e-04 Epoch 202/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9300 - loss: 0.1852 - val_accuracy: 0.9453 - val_loss: 0.1550 - learning_rate: 1.0000e-04 Epoch 203/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9239 - loss: 0.1978 - val_accuracy: 0.9453 - val_loss: 0.1556 - learning_rate: 1.0000e-04 Epoch 204/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9092 - loss: 0.2089 - val_accuracy: 0.9453 - val_loss: 0.1556 - learning_rate: 1.0000e-04 Epoch 205/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9197 - loss: 0.2034 - val_accuracy: 0.9432 - val_loss: 0.1556 - learning_rate: 1.0000e-04 Epoch 206/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9230 - loss: 0.1983 - val_accuracy: 0.9453 - val_loss: 0.1541 - learning_rate: 1.0000e-04 Epoch 207/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9280 - loss: 0.1871 - val_accuracy: 0.9453 - val_loss: 0.1540 - learning_rate: 1.0000e-04 Epoch 208/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9266 - loss: 0.1780 - val_accuracy: 0.9464 - val_loss: 0.1540 - learning_rate: 1.0000e-04 Epoch 209/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9156 - loss: 0.2090 - val_accuracy: 0.9464 - val_loss: 0.1538 - learning_rate: 1.0000e-04 Epoch 210/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9280 - loss: 0.1931 - val_accuracy: 0.9432 - val_loss: 0.1568 - learning_rate: 1.0000e-04 Epoch 211/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9183 - loss: 0.1914 - val_accuracy: 0.9443 - val_loss: 0.1552 - learning_rate: 1.0000e-04 Epoch 212/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9287 - loss: 0.1788 - val_accuracy: 0.9443 - val_loss: 0.1553 - learning_rate: 1.0000e-04 Epoch 213/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9238 - loss: 0.1844 - val_accuracy: 0.9443 - val_loss: 0.1548 - learning_rate: 1.0000e-04 Epoch 214/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9227 - loss: 0.1843 - val_accuracy: 0.9443 - val_loss: 0.1549 - learning_rate: 1.0000e-04 Epoch 215/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9268 - loss: 0.1873 - val_accuracy: 0.9411 - val_loss: 0.1581 - learning_rate: 1.0000e-04 Epoch 216/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9197 - loss: 0.1970 - val_accuracy: 0.9453 - val_loss: 0.1551 - learning_rate: 1.0000e-04 Epoch 217/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9203 - loss: 0.1912 - val_accuracy: 0.9453 - val_loss: 0.1538 - learning_rate: 1.0000e-04 Epoch 218/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9174 - loss: 0.1911 - val_accuracy: 0.9400 - val_loss: 0.1536 - learning_rate: 1.0000e-04 Epoch 219/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9263 - loss: 0.1977 - val_accuracy: 0.9453 - val_loss: 0.1548 - learning_rate: 1.0000e-04 Epoch 220/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9232 - loss: 0.1916 - val_accuracy: 0.9464 - val_loss: 0.1530 - learning_rate: 1.0000e-04 Epoch 221/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9294 - loss: 0.1822 - val_accuracy: 0.9464 - val_loss: 0.1531 - learning_rate: 1.0000e-04 Epoch 222/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9238 - loss: 0.1974 - val_accuracy: 0.9464 - val_loss: 0.1529 - learning_rate: 1.0000e-04 Epoch 223/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9223 - loss: 0.1927 - val_accuracy: 0.9453 - val_loss: 0.1540 - learning_rate: 1.0000e-04 Epoch 224/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9148 - loss: 0.2011 - val_accuracy: 0.9464 - val_loss: 0.1532 - learning_rate: 1.0000e-04 Epoch 225/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9128 - loss: 0.1980 - val_accuracy: 0.9453 - val_loss: 0.1538 - learning_rate: 1.0000e-04 Epoch 226/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9211 - loss: 0.1953 - val_accuracy: 0.9453 - val_loss: 0.1522 - learning_rate: 1.0000e-04 Epoch 227/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9218 - loss: 0.1932 - val_accuracy: 0.9453 - val_loss: 0.1532 - learning_rate: 1.0000e-04 Epoch 228/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9200 - loss: 0.2027 - val_accuracy: 0.9453 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 229/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9201 - loss: 0.1949 - val_accuracy: 0.9453 - val_loss: 0.1526 - learning_rate: 1.0000e-04 Epoch 230/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9315 - loss: 0.1914 - val_accuracy: 0.9443 - val_loss: 0.1530 - learning_rate: 1.0000e-04 Epoch 231/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9224 - loss: 0.1841 - val_accuracy: 0.9464 - val_loss: 0.1536 - learning_rate: 1.0000e-04 Epoch 232/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9151 - loss: 0.1994 - val_accuracy: 0.9453 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 233/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9187 - loss: 0.1981 - val_accuracy: 0.9453 - val_loss: 0.1549 - learning_rate: 1.0000e-04 Epoch 234/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9211 - loss: 0.2006 - val_accuracy: 0.9443 - val_loss: 0.1554 - learning_rate: 1.0000e-04 Epoch 235/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9140 - loss: 0.1944 - val_accuracy: 0.9453 - val_loss: 0.1541 - learning_rate: 1.0000e-04 Epoch 236/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9230 - loss: 0.1957 - val_accuracy: 0.9464 - val_loss: 0.1535 - learning_rate: 1.0000e-04 Epoch 237/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9273 - loss: 0.1906 - val_accuracy: 0.9443 - val_loss: 0.1534 - learning_rate: 1.0000e-04 Epoch 238/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9213 - loss: 0.1910 - val_accuracy: 0.9443 - val_loss: 0.1543 - learning_rate: 1.0000e-04 Epoch 239/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9167 - loss: 0.2033 - val_accuracy: 0.9443 - val_loss: 0.1546 - learning_rate: 1.0000e-04 Epoch 240/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9318 - loss: 0.1785 - val_accuracy: 0.9453 - val_loss: 0.1553 - learning_rate: 1.0000e-04 Epoch 241/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9181 - loss: 0.2020 - val_accuracy: 0.9443 - val_loss: 0.1551 - learning_rate: 1.0000e-04 Epoch 242/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9214 - loss: 0.1843 - val_accuracy: 0.9443 - val_loss: 0.1557 - learning_rate: 1.0000e-04 Epoch 243/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9153 - loss: 0.2000 - val_accuracy: 0.9453 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 244/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9301 - loss: 0.1786 - val_accuracy: 0.9453 - val_loss: 0.1531 - learning_rate: 1.0000e-04 Epoch 245/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9329 - loss: 0.1682 - val_accuracy: 0.9400 - val_loss: 0.1538 - learning_rate: 1.0000e-04 Epoch 246/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9168 - loss: 0.1966 - val_accuracy: 0.9453 - val_loss: 0.1546 - learning_rate: 1.0000e-04 Epoch 247/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9224 - loss: 0.1840 - val_accuracy: 0.9464 - val_loss: 0.1542 - learning_rate: 1.0000e-04 Epoch 248/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9233 - loss: 0.1908 - val_accuracy: 0.9443 - val_loss: 0.1535 - learning_rate: 1.0000e-04 Epoch 249/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9098 - loss: 0.1958 - val_accuracy: 0.9411 - val_loss: 0.1553 - learning_rate: 1.0000e-04 Epoch 250/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9258 - loss: 0.1918 - val_accuracy: 0.9453 - val_loss: 0.1531 - learning_rate: 1.0000e-04 Epoch 251/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9232 - loss: 0.1855 - val_accuracy: 0.9432 - val_loss: 0.1539 - learning_rate: 1.0000e-04 Epoch 252/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9136 - loss: 0.2023 - val_accuracy: 0.9443 - val_loss: 0.1541 - learning_rate: 1.0000e-04 Epoch 253/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9251 - loss: 0.1823 - val_accuracy: 0.9443 - val_loss: 0.1534 - learning_rate: 1.0000e-04 Epoch 254/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9200 - loss: 0.1925 - val_accuracy: 0.9453 - val_loss: 0.1536 - learning_rate: 1.0000e-04 Epoch 255/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9223 - loss: 0.1892 - val_accuracy: 0.9432 - val_loss: 0.1536 - learning_rate: 1.0000e-04 Epoch 256/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9240 - loss: 0.1859 - val_accuracy: 0.9443 - val_loss: 0.1531 - learning_rate: 1.0000e-04 Epoch 257/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9182 - loss: 0.1976 - val_accuracy: 0.9411 - val_loss: 0.1568 - learning_rate: 1.0000e-04 Epoch 258/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9241 - loss: 0.1955 - val_accuracy: 0.9443 - val_loss: 0.1525 - learning_rate: 1.0000e-04 Epoch 259/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9227 - loss: 0.1935 - val_accuracy: 0.9443 - val_loss: 0.1520 - learning_rate: 1.0000e-04 Epoch 260/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9243 - loss: 0.1878 - val_accuracy: 0.9443 - val_loss: 0.1546 - learning_rate: 1.0000e-04 Epoch 261/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9211 - loss: 0.1880 - val_accuracy: 0.9443 - val_loss: 0.1532 - learning_rate: 1.0000e-04 Epoch 262/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9198 - loss: 0.2032 - val_accuracy: 0.9453 - val_loss: 0.1533 - learning_rate: 1.0000e-04 Epoch 263/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9184 - loss: 0.2010 - val_accuracy: 0.9453 - val_loss: 0.1538 - learning_rate: 1.0000e-04 Epoch 264/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9236 - loss: 0.1811 - val_accuracy: 0.9453 - val_loss: 0.1544 - learning_rate: 1.0000e-04 Epoch 265/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9229 - loss: 0.1932 - val_accuracy: 0.9453 - val_loss: 0.1530 - learning_rate: 1.0000e-04 Epoch 266/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9228 - loss: 0.1971 - val_accuracy: 0.9453 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 267/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9155 - loss: 0.1997 - val_accuracy: 0.9453 - val_loss: 0.1541 - learning_rate: 1.0000e-04 Epoch 268/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9291 - loss: 0.1829 - val_accuracy: 0.9453 - val_loss: 0.1539 - learning_rate: 1.0000e-04 Epoch 269/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9182 - loss: 0.1951 - val_accuracy: 0.9453 - val_loss: 0.1542 - learning_rate: 1.0000e-04 Epoch 270/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9207 - loss: 0.1916 - val_accuracy: 0.9453 - val_loss: 0.1540 - learning_rate: 1.0000e-04 Epoch 271/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9199 - loss: 0.2000 - val_accuracy: 0.9453 - val_loss: 0.1542 - learning_rate: 1.0000e-04 Epoch 272/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9162 - loss: 0.1899 - val_accuracy: 0.9464 - val_loss: 0.1528 - learning_rate: 1.0000e-04 Epoch 273/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9271 - loss: 0.1780 - val_accuracy: 0.9443 - val_loss: 0.1531 - learning_rate: 1.0000e-04 Epoch 274/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9191 - loss: 0.2044 - val_accuracy: 0.9443 - val_loss: 0.1542 - learning_rate: 1.0000e-04 Epoch 275/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9215 - loss: 0.1945 - val_accuracy: 0.9464 - val_loss: 0.1522 - learning_rate: 1.0000e-04 Epoch 276/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9211 - loss: 0.1983 - val_accuracy: 0.9453 - val_loss: 0.1530 - learning_rate: 1.0000e-04 Epoch 277/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9154 - loss: 0.2079 - val_accuracy: 0.9453 - val_loss: 0.1529 - learning_rate: 1.0000e-04 Epoch 278/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9193 - loss: 0.1954 - val_accuracy: 0.9453 - val_loss: 0.1523 - learning_rate: 1.0000e-04 Epoch 279/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9270 - loss: 0.1859 - val_accuracy: 0.9453 - val_loss: 0.1524 - learning_rate: 1.0000e-04 Epoch 280/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9261 - loss: 0.1804 - val_accuracy: 0.9453 - val_loss: 0.1534 - learning_rate: 1.0000e-04 Epoch 281/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9275 - loss: 0.1824 - val_accuracy: 0.9453 - val_loss: 0.1525 - learning_rate: 1.0000e-04 Epoch 282/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9241 - loss: 0.1861 - val_accuracy: 0.9453 - val_loss: 0.1528 - learning_rate: 1.0000e-04 Epoch 283/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9265 - loss: 0.1910 - val_accuracy: 0.9453 - val_loss: 0.1521 - learning_rate: 1.0000e-04 Epoch 284/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9284 - loss: 0.1790 - val_accuracy: 0.9453 - val_loss: 0.1529 - learning_rate: 1.0000e-04 Epoch 285/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9208 - loss: 0.1961 - val_accuracy: 0.9443 - val_loss: 0.1555 - learning_rate: 1.0000e-04 Epoch 286/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9234 - loss: 0.1964 - val_accuracy: 0.9464 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 287/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9217 - loss: 0.1863 - val_accuracy: 0.9443 - val_loss: 0.1568 - learning_rate: 1.0000e-04 Epoch 288/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9292 - loss: 0.1801 - val_accuracy: 0.9464 - val_loss: 0.1541 - learning_rate: 1.0000e-04 Epoch 289/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9317 - loss: 0.1917 - val_accuracy: 0.9453 - val_loss: 0.1534 - learning_rate: 1.0000e-04 Epoch 290/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 32ms/step - accuracy: 0.9222 - loss: 0.1916 - val_accuracy: 0.9464 - val_loss: 0.1526 - learning_rate: 1.0000e-04 Epoch 291/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9209 - loss: 0.2012 - val_accuracy: 0.9443 - val_loss: 0.1555 - learning_rate: 1.0000e-04 Epoch 292/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 33ms/step - accuracy: 0.9198 - loss: 0.1974 - val_accuracy: 0.9443 - val_loss: 0.1544 - learning_rate: 1.0000e-04 Epoch 293/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9289 - loss: 0.1891 - val_accuracy: 0.9453 - val_loss: 0.1545 - learning_rate: 1.0000e-04 Epoch 294/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 7s 33ms/step - accuracy: 0.9305 - loss: 0.1902 - val_accuracy: 0.9443 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 295/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9186 - loss: 0.1976 - val_accuracy: 0.9443 - val_loss: 0.1537 - learning_rate: 1.0000e-04 Epoch 296/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9203 - loss: 0.1922 - val_accuracy: 0.9464 - val_loss: 0.1523 - learning_rate: 1.0000e-04 Epoch 297/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9233 - loss: 0.1924 - val_accuracy: 0.9443 - val_loss: 0.1532 - learning_rate: 1.0000e-04 Epoch 298/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 35ms/step - accuracy: 0.9255 - loss: 0.1944 - val_accuracy: 0.9443 - val_loss: 0.1535 - learning_rate: 1.0000e-04 Epoch 299/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 8s 34ms/step - accuracy: 0.9284 - loss: 0.1865 - val_accuracy: 0.9443 - val_loss: 0.1542 - learning_rate: 1.0000e-04 Epoch 299: early stopping Restoring model weights from the end of the best epoch: 259.
# Create dataframe of model fit history.
mod_cnn_hist_df = pd.DataFrame().from_dict(mod_cnn_hist.history, orient = 'columns')
plot_TF_training_history(mod_cnn_hist_df, 'CNN - Training History')
This looks great, we can see how the model stabilizes and converges nicely without overfitting, still gaining slight improvements all the way until epoch 259. During model building and tuning it was unclear if batch normalization layers were necessary in this network, but in the end it didn't seem necessary.
Looking at the training loss line we can see how much the loss oscillates until around 100 epochs this is likely due to the random image augmentations, helping create a more generalizable and robust model.
y_pred_val_cnn_proba = mod_cnn.predict(np.stack(X_val.norm_flat.apply(flat_to_array)), verbose = "auto", callbacks = None)
y_pred_val_cnn = y_pred_val_cnn_proba.argmax(axis = 1)
accuracy_score(y_val, y_pred_val_cnn)
30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 17ms/step
0.9442658092175777
The model achieves a 94.43% accuracy score on the validation set.
y_pred_cnn_proba = mod_cnn.predict(np.stack(X_test.norm_flat.apply(flat_to_array)), verbose = "auto", callbacks = None)
y_pred_cnn = y_pred_cnn_proba.argmax(axis = 1)
20/20 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step
Train the same model but use the training set that is not augmented/transformed for comparison at the end. Muted outputs for clarity.
mod_cnn_no_aug = build_cnn()
# Callbacks to use (same as above):
early_stop = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 40, verbose = 1, restore_best_weights = True)
reduce_lr_plateau = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, patience = 5, cooldown = 8, min_lr = 0.0001, verbose = 0)
optimizer_param = tf.keras.optimizers.SGD(learning_rate = 0.01)
mod_cnn_no_aug.compile(loss = loss_fx,
optimizer = optimizer_param,
metrics = ['accuracy'])
mod_cnn_no_aug_hist = mod_cnn_no_aug.fit(ds_train,
batch_size = BATCH_SIZE,
epochs = n_epochs,
verbose = 0,
callbacks = [early_stop, reduce_lr_plateau],
validation_split = 0.0,
validation_data = ds_val,
shuffle = True,
class_weight = None,
sample_weight = None,
initial_epoch = 0,
steps_per_epoch = None,
validation_steps = None,
validation_batch_size = None,
validation_freq = val_freq)
y_pred_val_cnn_no_aug_proba = mod_cnn_no_aug.predict(ds_val, verbose = 0, callbacks = None)
y_pred_val_cnn_no_aug = y_pred_val_cnn_no_aug_proba.argmax(axis = 1)
y_pred_cnn_no_aug_proba = mod_cnn_no_aug.predict(np.stack(X_test.norm_flat.apply(flat_to_array)), verbose = 0, callbacks = None)
y_pred_cnn_no_aug = y_pred_cnn_no_aug_proba.argmax(axis = 1)
Epoch 68: early stopping Restoring model weights from the end of the best epoch: 28.
Interestingly the model without augmented training images stops training and reverts back to weights from epoch 28.
Next, I would like to try to implement a model that is tried and tested on image classification tasks-ResNet50- which is the perfect mix of depth and efficiency here. Anything deeper would likely be overkill since the images input are only 56x56.
The benefit of the ResNet architecture is within it's name, Residual connections, sometimes called skip connections. These connections connect/concatenate outputs of earlier layers to inputs of later layers, skipping over a number of layers (in this case 2). This effectively combats the issue of vanishing gradients by allowing the network to skip over connections that hurt the performance of the classifier.
This model below was largely built from the source code for TensorFlow's ResNet50 from https://github.com/tensorflow/models/tree/master/official/vision and elsewhere. Slight alterations were made to the original architecture but it is mostly the same. Notably, I opted for a smaller kernel size in the first layer as this dataset's images are already very small. There are pre-trained versions of ResNet out there (and within Tensorflow for that matter), but I will build and train a new one for a more equal comparison between this and my previous, much shallower model.
# Inspired by ResNet 50 https://github.com/tensorflow/models/tree/master/official/vision
# Residual function to set skip connections. Downsample to reduce dimension.
def residual_block(x, filters, downsample = False):
shortcut = x
strides = (2, 2) if downsample else (1, 1)
# First convolution layer.
x = tf.keras.layers.Conv2D(filters, kernel_size = (3, 3), strides = strides, padding = 'same')(x)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.ReLU()(x)
# Second convolution layer.
x = tf.keras.layers.Conv2D(filters, kernel_size = (3, 3), padding = 'same')(x)
x = tf.keras.layers.BatchNormalization()(x)
# Add convolution if downsample == True.
if downsample:
shortcut = tf.keras.layers.Conv2D(filters, kernel_size = (1, 1), strides = (2, 2), padding = 'same')(shortcut)
shortcut = tf.keras.layers.BatchNormalization()(shortcut)
# Add the shortcut (input) to the output.
x = tf.keras.layers.Add()([x, shortcut])
x = tf.keras.layers.ReLU()(x)
return x
# Build model.
def build_dcnn(input_shape = (img_size, img_size, 3)):
inputs = tf.keras.Input(shape = input_shape)
# Initial convolutional layer. Reduced kernel size here from default 7,7 because the input images are small.
x = tf.keras.layers.Conv2D(64, kernel_size = (5, 5), strides = (2, 2), padding = 'same')(inputs)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.ReLU()(x)
x = tf.keras.layers.MaxPooling2D(pool_size = (3, 3), strides = (2, 2), padding = 'same')(x)
# First residual block (3 blocks).
for _ in range(3):
x = residual_block(x, 64)
# Second residual block (4 blocks).
x = residual_block(x, 128, downsample = True)
for _ in range(3):
x = residual_block(x, 128)
# Third residual block (6 blocks).
x = residual_block(x, 256, downsample = True)
for _ in range(5):
x = residual_block(x, 256)
# Fourth residual block (3 blocks).
x = residual_block(x, 512, downsample = True)
for _ in range(2):
x = residual_block(x, 512)
# Global average pooling layer and final output prediction.
x = tf.keras.layers.GlobalAveragePooling2D()(x)
outputs = tf.keras.layers.Dense(2, name = 'Predictions')(x)
# Create model.
model = tf.keras.Model(inputs, outputs)
return model
# Finally, build the model.
mod_dcnn = build_dcnn(input_shape=(img_size, img_size, 3))
mod_dcnn.summary()
Model: "functional_5"
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ Connected to ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ input_layer_1 │ (None, 56, 56, 3) │ 0 │ - │ │ (InputLayer) │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_10 (Conv2D) │ (None, 28, 28, │ 4,864 │ input_layer_1[0]… │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalization │ (None, 28, 28, │ 256 │ conv2d_10[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_18 (ReLU) │ (None, 28, 28, │ 0 │ batch_normalizat… │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ max_pooling2d_4 │ (None, 14, 14, │ 0 │ re_lu_18[0][0] │ │ (MaxPooling2D) │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_11 (Conv2D) │ (None, 14, 14, │ 36,928 │ max_pooling2d_4[… │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 14, 14, │ 256 │ conv2d_11[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_19 (ReLU) │ (None, 14, 14, │ 0 │ batch_normalizat… │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_12 (Conv2D) │ (None, 14, 14, │ 36,928 │ re_lu_19[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 14, 14, │ 256 │ conv2d_12[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add (Add) │ (None, 14, 14, │ 0 │ batch_normalizat… │ │ │ 64) │ │ max_pooling2d_4[… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_20 (ReLU) │ (None, 14, 14, │ 0 │ add[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_13 (Conv2D) │ (None, 14, 14, │ 36,928 │ re_lu_20[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 14, 14, │ 256 │ conv2d_13[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_21 (ReLU) │ (None, 14, 14, │ 0 │ batch_normalizat… │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_14 (Conv2D) │ (None, 14, 14, │ 36,928 │ re_lu_21[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 14, 14, │ 256 │ conv2d_14[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_1 (Add) │ (None, 14, 14, │ 0 │ batch_normalizat… │ │ │ 64) │ │ re_lu_20[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_22 (ReLU) │ (None, 14, 14, │ 0 │ add_1[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_15 (Conv2D) │ (None, 14, 14, │ 36,928 │ re_lu_22[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 14, 14, │ 256 │ conv2d_15[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_23 (ReLU) │ (None, 14, 14, │ 0 │ batch_normalizat… │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_16 (Conv2D) │ (None, 14, 14, │ 36,928 │ re_lu_23[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 14, 14, │ 256 │ conv2d_16[0][0] │ │ (BatchNormalizatio… │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_2 (Add) │ (None, 14, 14, │ 0 │ batch_normalizat… │ │ │ 64) │ │ re_lu_22[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_24 (ReLU) │ (None, 14, 14, │ 0 │ add_2[0][0] │ │ │ 64) │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_17 (Conv2D) │ (None, 7, 7, 128) │ 73,856 │ re_lu_24[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_17[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_25 (ReLU) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_18 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_25[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_19 (Conv2D) │ (None, 7, 7, 128) │ 8,320 │ re_lu_24[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_18[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_19[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_3 (Add) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ │ │ │ │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_26 (ReLU) │ (None, 7, 7, 128) │ 0 │ add_3[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_20 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_26[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_20[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_27 (ReLU) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_21 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_27[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_21[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_4 (Add) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_26[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_28 (ReLU) │ (None, 7, 7, 128) │ 0 │ add_4[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_22 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_28[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_22[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_29 (ReLU) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_23 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_29[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_23[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_5 (Add) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_28[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_30 (ReLU) │ (None, 7, 7, 128) │ 0 │ add_5[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_24 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_30[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_24[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_31 (ReLU) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_25 (Conv2D) │ (None, 7, 7, 128) │ 147,584 │ re_lu_31[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 7, 7, 128) │ 512 │ conv2d_25[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_6 (Add) │ (None, 7, 7, 128) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_30[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_32 (ReLU) │ (None, 7, 7, 128) │ 0 │ add_6[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_26 (Conv2D) │ (None, 4, 4, 256) │ 295,168 │ re_lu_32[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_26[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_33 (ReLU) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_27 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_33[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_28 (Conv2D) │ (None, 4, 4, 256) │ 33,024 │ re_lu_32[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_27[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_28[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_7 (Add) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ │ │ │ │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_34 (ReLU) │ (None, 4, 4, 256) │ 0 │ add_7[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_29 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_34[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_29[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_35 (ReLU) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_30 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_35[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_30[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_8 (Add) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_34[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_36 (ReLU) │ (None, 4, 4, 256) │ 0 │ add_8[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_31 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_36[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_31[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_37 (ReLU) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_32 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_37[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_32[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_9 (Add) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_36[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_38 (ReLU) │ (None, 4, 4, 256) │ 0 │ add_9[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_33 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_38[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_33[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_39 (ReLU) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_34 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_39[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_34[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_10 (Add) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_38[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_40 (ReLU) │ (None, 4, 4, 256) │ 0 │ add_10[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_35 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_40[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_35[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_41 (ReLU) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_36 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_41[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_36[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_11 (Add) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_40[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_42 (ReLU) │ (None, 4, 4, 256) │ 0 │ add_11[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_37 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_42[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_37[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_43 (ReLU) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_38 (Conv2D) │ (None, 4, 4, 256) │ 590,080 │ re_lu_43[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 4, 4, 256) │ 1,024 │ conv2d_38[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_12 (Add) │ (None, 4, 4, 256) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_42[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_44 (ReLU) │ (None, 4, 4, 256) │ 0 │ add_12[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_39 (Conv2D) │ (None, 2, 2, 512) │ 1,180,160 │ re_lu_44[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_39[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_45 (ReLU) │ (None, 2, 2, 512) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_40 (Conv2D) │ (None, 2, 2, 512) │ 2,359,808 │ re_lu_45[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_41 (Conv2D) │ (None, 2, 2, 512) │ 131,584 │ re_lu_44[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_40[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_41[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_13 (Add) │ (None, 2, 2, 512) │ 0 │ batch_normalizat… │ │ │ │ │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_46 (ReLU) │ (None, 2, 2, 512) │ 0 │ add_13[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_42 (Conv2D) │ (None, 2, 2, 512) │ 2,359,808 │ re_lu_46[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_42[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_47 (ReLU) │ (None, 2, 2, 512) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_43 (Conv2D) │ (None, 2, 2, 512) │ 2,359,808 │ re_lu_47[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_43[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_14 (Add) │ (None, 2, 2, 512) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_46[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_48 (ReLU) │ (None, 2, 2, 512) │ 0 │ add_14[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_44 (Conv2D) │ (None, 2, 2, 512) │ 2,359,808 │ re_lu_48[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_44[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_49 (ReLU) │ (None, 2, 2, 512) │ 0 │ batch_normalizat… │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ conv2d_45 (Conv2D) │ (None, 2, 2, 512) │ 2,359,808 │ re_lu_49[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ batch_normalizatio… │ (None, 2, 2, 512) │ 2,048 │ conv2d_45[0][0] │ │ (BatchNormalizatio… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ add_15 (Add) │ (None, 2, 2, 512) │ 0 │ batch_normalizat… │ │ │ │ │ re_lu_48[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ re_lu_50 (ReLU) │ (None, 2, 2, 512) │ 0 │ add_15[0][0] │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ global_average_poo… │ (None, 512) │ 0 │ re_lu_50[0][0] │ │ (GlobalAveragePool… │ │ │ │ ├─────────────────────┼───────────────────┼────────────┼───────────────────┤ │ Predictions (Dense) │ (None, 2) │ 1,026 │ global_average_p… │ └─────────────────────┴───────────────────┴────────────┴───────────────────┘
Total params: 21,306,626 (81.28 MB)
Trainable params: 21,289,602 (81.21 MB)
Non-trainable params: 17,024 (66.50 KB)
visualkeras.layered_view(mod_dcnn, legend = True)
As expected, that's a lot of layers. Also as another example of the power of stacked 3x3 layers, note the number of trainable parameters here - 21,289,602. That's almost the same amount as my the previous model's architecture with 5x5 and 7x7 kernels!
As with all other models the same callbacks are used, but were slightly modified to suit this network.
# Using similar loss, optimizer, and callbacks here.
loss_fx = tf.keras.losses.SparseCategoricalCrossentropy(from_logits = True) # Integer labels.
optimizer_param = 'sgd'
val_freq = 1
n_epochs = 300
# Callbacks to use:
early_stop = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 30, verbose = 1, restore_best_weights = True)
reduce_lr_plateau = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss',
factor = 0.5,
patience = 5,
cooldown = 8,
min_lr = 0.0001,
verbose = 1)
mod_dcnn.compile(loss = loss_fx,
optimizer = optimizer_param,
metrics = ['accuracy'])
mod_dcnn_hist = mod_dcnn.fit(ds_train_transformed,
batch_size = BATCH_SIZE,
epochs = n_epochs,
verbose = "auto",
callbacks = [early_stop, reduce_lr_plateau],
validation_split = 0.0,
validation_data = ds_val,
shuffle = True,
class_weight = None,
sample_weight = None,
initial_epoch = 0,
steps_per_epoch = None,
validation_steps = None,
validation_batch_size = None,
validation_freq = val_freq)
Epoch 1/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 58s 240ms/step - accuracy: 0.5159 - loss: 1.3220 - val_accuracy: 0.6120 - val_loss: 0.6770 - learning_rate: 0.0100 Epoch 2/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.6021 - loss: 0.8050 - val_accuracy: 0.7138 - val_loss: 0.5994 - learning_rate: 0.0100 Epoch 3/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 234ms/step - accuracy: 0.6490 - loss: 0.6986 - val_accuracy: 0.7353 - val_loss: 0.5611 - learning_rate: 0.0100 Epoch 4/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.7088 - loss: 0.5964 - val_accuracy: 0.7771 - val_loss: 0.4710 - learning_rate: 0.0100 Epoch 5/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.7268 - loss: 0.5578 - val_accuracy: 0.7953 - val_loss: 0.4342 - learning_rate: 0.0100 Epoch 6/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.7410 - loss: 0.5364 - val_accuracy: 0.7599 - val_loss: 0.5030 - learning_rate: 0.0100 Epoch 7/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.7577 - loss: 0.4890 - val_accuracy: 0.8124 - val_loss: 0.4142 - learning_rate: 0.0100 Epoch 8/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 234ms/step - accuracy: 0.7613 - loss: 0.4814 - val_accuracy: 0.7320 - val_loss: 0.5295 - learning_rate: 0.0100 Epoch 9/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 234ms/step - accuracy: 0.7802 - loss: 0.4428 - val_accuracy: 0.8049 - val_loss: 0.4384 - learning_rate: 0.0100 Epoch 10/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 234ms/step - accuracy: 0.8045 - loss: 0.4365 - val_accuracy: 0.8435 - val_loss: 0.3543 - learning_rate: 0.0100 Epoch 11/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.8004 - loss: 0.4236 - val_accuracy: 0.8703 - val_loss: 0.3057 - learning_rate: 0.0100 Epoch 12/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 245ms/step - accuracy: 0.8095 - loss: 0.4275 - val_accuracy: 0.7406 - val_loss: 0.5253 - learning_rate: 0.0100 Epoch 13/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.8204 - loss: 0.4064 - val_accuracy: 0.8735 - val_loss: 0.2879 - learning_rate: 0.0100 Epoch 14/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 240ms/step - accuracy: 0.8389 - loss: 0.3760 - val_accuracy: 0.8725 - val_loss: 0.3099 - learning_rate: 0.0100 Epoch 15/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.8382 - loss: 0.3619 - val_accuracy: 0.8242 - val_loss: 0.4151 - learning_rate: 0.0100 Epoch 16/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.8570 - loss: 0.3319 - val_accuracy: 0.7513 - val_loss: 0.5177 - learning_rate: 0.0100 Epoch 17/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.8465 - loss: 0.3442 - val_accuracy: 0.8574 - val_loss: 0.3644 - learning_rate: 0.0100 Epoch 18/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 228ms/step - accuracy: 0.8431 - loss: 0.3518 Epoch 18: ReduceLROnPlateau reducing learning rate to 0.004999999888241291. 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 240ms/step - accuracy: 0.8432 - loss: 0.3517 - val_accuracy: 0.8135 - val_loss: 0.4336 - learning_rate: 0.0100 Epoch 19/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.8719 - loss: 0.2980 - val_accuracy: 0.8939 - val_loss: 0.2597 - learning_rate: 0.0050 Epoch 20/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.8707 - loss: 0.2879 - val_accuracy: 0.8950 - val_loss: 0.2823 - learning_rate: 0.0050 Epoch 21/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 236ms/step - accuracy: 0.8799 - loss: 0.2852 - val_accuracy: 0.9164 - val_loss: 0.2090 - learning_rate: 0.0050 Epoch 22/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.8847 - loss: 0.2809 - val_accuracy: 0.9057 - val_loss: 0.2552 - learning_rate: 0.0050 Epoch 23/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.8870 - loss: 0.2738 - val_accuracy: 0.8800 - val_loss: 0.2736 - learning_rate: 0.0050 Epoch 24/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.8944 - loss: 0.2663 - val_accuracy: 0.9025 - val_loss: 0.2290 - learning_rate: 0.0050 Epoch 25/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.8937 - loss: 0.2671 - val_accuracy: 0.8950 - val_loss: 0.2500 - learning_rate: 0.0050 Epoch 26/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.8953 - loss: 0.2642 - val_accuracy: 0.9110 - val_loss: 0.2217 - learning_rate: 0.0050 Epoch 27/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.9016 - loss: 0.2425 - val_accuracy: 0.8992 - val_loss: 0.2435 - learning_rate: 0.0050 Epoch 28/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.8895 - loss: 0.2738 - val_accuracy: 0.9089 - val_loss: 0.2514 - learning_rate: 0.0050 Epoch 29/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.8967 - loss: 0.2497 - val_accuracy: 0.9175 - val_loss: 0.2059 - learning_rate: 0.0050 Epoch 30/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.8842 - loss: 0.2688 - val_accuracy: 0.8585 - val_loss: 0.3477 - learning_rate: 0.0050 Epoch 31/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.9110 - loss: 0.2226 - val_accuracy: 0.9121 - val_loss: 0.2395 - learning_rate: 0.0050 Epoch 32/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.8957 - loss: 0.2386 - val_accuracy: 0.8800 - val_loss: 0.2733 - learning_rate: 0.0050 Epoch 33/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.9070 - loss: 0.2415 - val_accuracy: 0.9164 - val_loss: 0.2105 - learning_rate: 0.0050 Epoch 34/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 227ms/step - accuracy: 0.9065 - loss: 0.2333 Epoch 34: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455. 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.9064 - loss: 0.2333 - val_accuracy: 0.8296 - val_loss: 0.3811 - learning_rate: 0.0050 Epoch 35/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.9049 - loss: 0.2200 - val_accuracy: 0.9271 - val_loss: 0.1888 - learning_rate: 0.0025 Epoch 36/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9166 - loss: 0.2086 - val_accuracy: 0.9250 - val_loss: 0.1979 - learning_rate: 0.0025 Epoch 37/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9098 - loss: 0.2190 - val_accuracy: 0.9078 - val_loss: 0.2182 - learning_rate: 0.0025 Epoch 38/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.9107 - loss: 0.2249 - val_accuracy: 0.9314 - val_loss: 0.1888 - learning_rate: 0.0025 Epoch 39/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9195 - loss: 0.2015 - val_accuracy: 0.9293 - val_loss: 0.1879 - learning_rate: 0.0025 Epoch 40/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.9189 - loss: 0.2037 - val_accuracy: 0.9303 - val_loss: 0.1782 - learning_rate: 0.0025 Epoch 41/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9242 - loss: 0.1932 - val_accuracy: 0.9282 - val_loss: 0.1821 - learning_rate: 0.0025 Epoch 42/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9173 - loss: 0.1997 - val_accuracy: 0.9282 - val_loss: 0.1796 - learning_rate: 0.0025 Epoch 43/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9232 - loss: 0.1973 - val_accuracy: 0.9293 - val_loss: 0.1809 - learning_rate: 0.0025 Epoch 44/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.9209 - loss: 0.1923 - val_accuracy: 0.8853 - val_loss: 0.3099 - learning_rate: 0.0025 Epoch 45/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 244ms/step - accuracy: 0.9233 - loss: 0.1947 - val_accuracy: 0.9196 - val_loss: 0.2263 - learning_rate: 0.0025 Epoch 46/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9084 - loss: 0.2117 - val_accuracy: 0.9346 - val_loss: 0.1766 - learning_rate: 0.0025 Epoch 47/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.9191 - loss: 0.1876 - val_accuracy: 0.9293 - val_loss: 0.1790 - learning_rate: 0.0025 Epoch 48/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.9171 - loss: 0.1813 - val_accuracy: 0.9207 - val_loss: 0.2091 - learning_rate: 0.0025 Epoch 49/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.9186 - loss: 0.1983 - val_accuracy: 0.9368 - val_loss: 0.1694 - learning_rate: 0.0025 Epoch 50/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.9243 - loss: 0.1881 - val_accuracy: 0.9314 - val_loss: 0.1777 - learning_rate: 0.0025 Epoch 51/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9324 - loss: 0.1726 - val_accuracy: 0.9014 - val_loss: 0.2521 - learning_rate: 0.0025 Epoch 52/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.9193 - loss: 0.1911 - val_accuracy: 0.9335 - val_loss: 0.1771 - learning_rate: 0.0025 Epoch 53/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9229 - loss: 0.1789 - val_accuracy: 0.9218 - val_loss: 0.2242 - learning_rate: 0.0025 Epoch 54/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 231ms/step - accuracy: 0.9405 - loss: 0.1582 Epoch 54: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228. 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.9405 - loss: 0.1582 - val_accuracy: 0.9389 - val_loss: 0.1799 - learning_rate: 0.0025 Epoch 55/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.9292 - loss: 0.1820 - val_accuracy: 0.9303 - val_loss: 0.1933 - learning_rate: 0.0012 Epoch 56/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.9353 - loss: 0.1626 - val_accuracy: 0.9357 - val_loss: 0.1613 - learning_rate: 0.0012 Epoch 57/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.9330 - loss: 0.1691 - val_accuracy: 0.9346 - val_loss: 0.1699 - learning_rate: 0.0012 Epoch 58/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 232ms/step - accuracy: 0.9313 - loss: 0.1723 - val_accuracy: 0.9325 - val_loss: 0.2126 - learning_rate: 0.0012 Epoch 59/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 234ms/step - accuracy: 0.9368 - loss: 0.1595 - val_accuracy: 0.9421 - val_loss: 0.1676 - learning_rate: 0.0012 Epoch 60/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.9356 - loss: 0.1692 - val_accuracy: 0.9368 - val_loss: 0.1806 - learning_rate: 0.0012 Epoch 61/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.9346 - loss: 0.1639 - val_accuracy: 0.9400 - val_loss: 0.1685 - learning_rate: 0.0012 Epoch 62/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 235ms/step - accuracy: 0.9293 - loss: 0.1634 - val_accuracy: 0.9357 - val_loss: 0.1912 - learning_rate: 0.0012 Epoch 63/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 52s 230ms/step - accuracy: 0.9307 - loss: 0.1719 - val_accuracy: 0.9346 - val_loss: 0.1623 - learning_rate: 0.0012 Epoch 64/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 53s 232ms/step - accuracy: 0.9440 - loss: 0.1510 - val_accuracy: 0.9421 - val_loss: 0.1583 - learning_rate: 0.0012 Epoch 65/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 240ms/step - accuracy: 0.9388 - loss: 0.1519 - val_accuracy: 0.9368 - val_loss: 0.1832 - learning_rate: 0.0012 Epoch 66/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 245ms/step - accuracy: 0.9415 - loss: 0.1522 - val_accuracy: 0.9260 - val_loss: 0.2171 - learning_rate: 0.0012 Epoch 67/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.9460 - loss: 0.1427 - val_accuracy: 0.9335 - val_loss: 0.1790 - learning_rate: 0.0012 Epoch 68/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 237ms/step - accuracy: 0.9333 - loss: 0.1567 - val_accuracy: 0.9411 - val_loss: 0.1793 - learning_rate: 0.0012 Epoch 69/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 229ms/step - accuracy: 0.9337 - loss: 0.1603 Epoch 69: ReduceLROnPlateau reducing learning rate to 0.0006249999860301614. 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9338 - loss: 0.1603 - val_accuracy: 0.9346 - val_loss: 0.1919 - learning_rate: 0.0012 Epoch 70/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 240ms/step - accuracy: 0.9346 - loss: 0.1541 - val_accuracy: 0.9378 - val_loss: 0.1620 - learning_rate: 6.2500e-04 Epoch 71/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9464 - loss: 0.1414 - val_accuracy: 0.9357 - val_loss: 0.1653 - learning_rate: 6.2500e-04 Epoch 72/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9308 - loss: 0.1532 - val_accuracy: 0.9368 - val_loss: 0.1639 - learning_rate: 6.2500e-04 Epoch 73/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9478 - loss: 0.1418 - val_accuracy: 0.9357 - val_loss: 0.1656 - learning_rate: 6.2500e-04 Epoch 74/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9423 - loss: 0.1571 - val_accuracy: 0.9207 - val_loss: 0.1945 - learning_rate: 6.2500e-04 Epoch 75/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 245ms/step - accuracy: 0.9487 - loss: 0.1415 - val_accuracy: 0.9389 - val_loss: 0.1745 - learning_rate: 6.2500e-04 Epoch 76/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9447 - loss: 0.1312 - val_accuracy: 0.9357 - val_loss: 0.1790 - learning_rate: 6.2500e-04 Epoch 77/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 238ms/step - accuracy: 0.9427 - loss: 0.1393 - val_accuracy: 0.9411 - val_loss: 0.1599 - learning_rate: 6.2500e-04 Epoch 78/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 245ms/step - accuracy: 0.9524 - loss: 0.1367 - val_accuracy: 0.9293 - val_loss: 0.1662 - learning_rate: 6.2500e-04 Epoch 79/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 240ms/step - accuracy: 0.9450 - loss: 0.1496 - val_accuracy: 0.9411 - val_loss: 0.1693 - learning_rate: 6.2500e-04 Epoch 80/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9459 - loss: 0.1295 - val_accuracy: 0.9303 - val_loss: 0.1984 - learning_rate: 6.2500e-04 Epoch 81/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 232ms/step - accuracy: 0.9487 - loss: 0.1411 Epoch 81: ReduceLROnPlateau reducing learning rate to 0.0003124999930150807. 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 245ms/step - accuracy: 0.9487 - loss: 0.1411 - val_accuracy: 0.9378 - val_loss: 0.1812 - learning_rate: 6.2500e-04 Epoch 82/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 246ms/step - accuracy: 0.9483 - loss: 0.1393 - val_accuracy: 0.9411 - val_loss: 0.1701 - learning_rate: 3.1250e-04 Epoch 83/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9460 - loss: 0.1446 - val_accuracy: 0.9378 - val_loss: 0.1759 - learning_rate: 3.1250e-04 Epoch 84/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 240ms/step - accuracy: 0.9503 - loss: 0.1326 - val_accuracy: 0.9378 - val_loss: 0.1694 - learning_rate: 3.1250e-04 Epoch 85/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 241ms/step - accuracy: 0.9454 - loss: 0.1365 - val_accuracy: 0.9389 - val_loss: 0.1686 - learning_rate: 3.1250e-04 Epoch 86/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 54s 239ms/step - accuracy: 0.9518 - loss: 0.1207 - val_accuracy: 0.9432 - val_loss: 0.1537 - learning_rate: 3.1250e-04 Epoch 87/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 244ms/step - accuracy: 0.9485 - loss: 0.1308 - val_accuracy: 0.9400 - val_loss: 0.1558 - learning_rate: 3.1250e-04 Epoch 88/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9413 - loss: 0.1566 - val_accuracy: 0.9443 - val_loss: 0.1584 - learning_rate: 3.1250e-04 Epoch 89/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9504 - loss: 0.1267 - val_accuracy: 0.9411 - val_loss: 0.1633 - learning_rate: 3.1250e-04 Epoch 90/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9491 - loss: 0.1356 - val_accuracy: 0.9411 - val_loss: 0.1576 - learning_rate: 3.1250e-04 Epoch 91/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 247ms/step - accuracy: 0.9436 - loss: 0.1392 - val_accuracy: 0.9432 - val_loss: 0.1540 - learning_rate: 3.1250e-04 Epoch 92/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9485 - loss: 0.1341 - val_accuracy: 0.9453 - val_loss: 0.1556 - learning_rate: 3.1250e-04 Epoch 93/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 230ms/step - accuracy: 0.9603 - loss: 0.1175 Epoch 93: ReduceLROnPlateau reducing learning rate to 0.00015624999650754035. 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.9603 - loss: 0.1176 - val_accuracy: 0.9389 - val_loss: 0.1587 - learning_rate: 3.1250e-04 Epoch 94/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 243ms/step - accuracy: 0.9565 - loss: 0.1160 - val_accuracy: 0.9421 - val_loss: 0.1549 - learning_rate: 1.5625e-04 Epoch 95/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 242ms/step - accuracy: 0.9471 - loss: 0.1452 - val_accuracy: 0.9421 - val_loss: 0.1574 - learning_rate: 1.5625e-04 Epoch 96/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 55s 244ms/step - accuracy: 0.9598 - loss: 0.1135 - val_accuracy: 0.9453 - val_loss: 0.1524 - learning_rate: 1.5625e-04 Epoch 97/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 247ms/step - accuracy: 0.9565 - loss: 0.1260 - val_accuracy: 0.9486 - val_loss: 0.1548 - learning_rate: 1.5625e-04 Epoch 98/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 250ms/step - accuracy: 0.9582 - loss: 0.1107 - val_accuracy: 0.9378 - val_loss: 0.1660 - learning_rate: 1.5625e-04 Epoch 99/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9475 - loss: 0.1307 - val_accuracy: 0.9368 - val_loss: 0.1635 - learning_rate: 1.5625e-04 Epoch 100/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9512 - loss: 0.1254 - val_accuracy: 0.9464 - val_loss: 0.1579 - learning_rate: 1.5625e-04 Epoch 101/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 246ms/step - accuracy: 0.9481 - loss: 0.1337 - val_accuracy: 0.9453 - val_loss: 0.1571 - learning_rate: 1.5625e-04 Epoch 102/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 250ms/step - accuracy: 0.9447 - loss: 0.1389 - val_accuracy: 0.9411 - val_loss: 0.1592 - learning_rate: 1.5625e-04 Epoch 103/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9584 - loss: 0.1148 - val_accuracy: 0.9368 - val_loss: 0.1688 - learning_rate: 1.5625e-04 Epoch 104/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 246ms/step - accuracy: 0.9482 - loss: 0.1348 - val_accuracy: 0.9464 - val_loss: 0.1584 - learning_rate: 1.5625e-04 Epoch 105/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 0s 238ms/step - accuracy: 0.9530 - loss: 0.1311 Epoch 105: ReduceLROnPlateau reducing learning rate to 0.0001. 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9530 - loss: 0.1311 - val_accuracy: 0.9443 - val_loss: 0.1575 - learning_rate: 1.5625e-04 Epoch 106/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9490 - loss: 0.1325 - val_accuracy: 0.9400 - val_loss: 0.1600 - learning_rate: 1.0000e-04 Epoch 107/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 247ms/step - accuracy: 0.9428 - loss: 0.1523 - val_accuracy: 0.9411 - val_loss: 0.1599 - learning_rate: 1.0000e-04 Epoch 108/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 246ms/step - accuracy: 0.9530 - loss: 0.1174 - val_accuracy: 0.9421 - val_loss: 0.1592 - learning_rate: 1.0000e-04 Epoch 109/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 251ms/step - accuracy: 0.9472 - loss: 0.1252 - val_accuracy: 0.9432 - val_loss: 0.1573 - learning_rate: 1.0000e-04 Epoch 110/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 250ms/step - accuracy: 0.9519 - loss: 0.1218 - val_accuracy: 0.9421 - val_loss: 0.1587 - learning_rate: 1.0000e-04 Epoch 111/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9573 - loss: 0.1242 - val_accuracy: 0.9432 - val_loss: 0.1586 - learning_rate: 1.0000e-04 Epoch 112/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 246ms/step - accuracy: 0.9521 - loss: 0.1141 - val_accuracy: 0.9432 - val_loss: 0.1585 - learning_rate: 1.0000e-04 Epoch 113/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 246ms/step - accuracy: 0.9518 - loss: 0.1186 - val_accuracy: 0.9378 - val_loss: 0.1669 - learning_rate: 1.0000e-04 Epoch 114/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9516 - loss: 0.1244 - val_accuracy: 0.9421 - val_loss: 0.1607 - learning_rate: 1.0000e-04 Epoch 115/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9554 - loss: 0.1124 - val_accuracy: 0.9453 - val_loss: 0.1595 - learning_rate: 1.0000e-04 Epoch 116/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 249ms/step - accuracy: 0.9466 - loss: 0.1312 - val_accuracy: 0.9378 - val_loss: 0.1639 - learning_rate: 1.0000e-04 Epoch 117/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9521 - loss: 0.1189 - val_accuracy: 0.9421 - val_loss: 0.1621 - learning_rate: 1.0000e-04 Epoch 118/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9528 - loss: 0.1155 - val_accuracy: 0.9421 - val_loss: 0.1604 - learning_rate: 1.0000e-04 Epoch 119/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 247ms/step - accuracy: 0.9549 - loss: 0.1122 - val_accuracy: 0.9400 - val_loss: 0.1599 - learning_rate: 1.0000e-04 Epoch 120/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 58s 254ms/step - accuracy: 0.9499 - loss: 0.1227 - val_accuracy: 0.9432 - val_loss: 0.1589 - learning_rate: 1.0000e-04 Epoch 121/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 253ms/step - accuracy: 0.9484 - loss: 0.1311 - val_accuracy: 0.9378 - val_loss: 0.1618 - learning_rate: 1.0000e-04 Epoch 122/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 253ms/step - accuracy: 0.9577 - loss: 0.1112 - val_accuracy: 0.9400 - val_loss: 0.1632 - learning_rate: 1.0000e-04 Epoch 123/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9546 - loss: 0.1122 - val_accuracy: 0.9443 - val_loss: 0.1570 - learning_rate: 1.0000e-04 Epoch 124/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 56s 248ms/step - accuracy: 0.9570 - loss: 0.1175 - val_accuracy: 0.9443 - val_loss: 0.1578 - learning_rate: 1.0000e-04 Epoch 125/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 252ms/step - accuracy: 0.9500 - loss: 0.1314 - val_accuracy: 0.9421 - val_loss: 0.1599 - learning_rate: 1.0000e-04 Epoch 126/300 227/227 ━━━━━━━━━━━━━━━━━━━━ 57s 252ms/step - accuracy: 0.9559 - loss: 0.1166 - val_accuracy: 0.9400 - val_loss: 0.1636 - learning_rate: 1.0000e-04 Epoch 126: early stopping Restoring model weights from the end of the best epoch: 96.
# Create dataframe of model fit history.
mod_dcnn_hist_df = pd.DataFrame().from_dict(mod_dcnn_hist.history, orient = 'columns')
plot_TF_training_history(mod_dcnn_hist_df, 'DCNN - Training History')
Impressive we can see that this model trains a little quicker than the previous one, ending at epoch 96. It also seems to just slightly overfit the training data but does utilize batch normalization between each convolution layer so the effects are mitigated.
Similarly we see a lot of oscillation in the performance in the beginning, eventually stabilizing around epoch 80.
y_pred_val_dcnn_proba = mod_dcnn.predict(np.stack(X_val.norm_flat.apply(flat_to_array)), verbose = "auto", callbacks = None)
y_pred_val_dcnn = y_pred_val_dcnn_proba.argmax(axis = 1)
accuracy_score(y_val, y_pred_val_dcnn)
30/30 ━━━━━━━━━━━━━━━━━━━━ 3s 92ms/step
0.9453376205787781
94.53 accuracy on the validation set, just barely beating out the previous model. It will be interesting to see how they compare on the test set.
y_pred_dcnn_proba = mod_dcnn.predict(np.stack(X_test.norm_flat.apply(flat_to_array)), verbose = "auto", callbacks = None)
y_pred_dcnn = y_pred_dcnn_proba.argmax(axis = 1)
20/20 ━━━━━━━━━━━━━━━━━━━━ 2s 81ms/step
Train the same model but use the training set that is not augmented/transformed for comparison at the end. Muted outputs for clarity.
Note: For a fair comparison, the learning rate of the optimizer was lowered from the default: 0.01 to 0.005 as the model was quickly overfitting the training data and the model was very unstable.
mod_dcnn_no_aug = build_dcnn(input_shape=(img_size, img_size, 3))
optimizer_param = tf.keras.optimizers.SGD(learning_rate = 0.005)
mod_dcnn_no_aug.compile(loss = loss_fx,
optimizer = optimizer_param,
metrics = ['accuracy'])
# Callbacks to use (same as above):
early_stop = tf.keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 30, verbose = 1, restore_best_weights = True)
reduce_lr_plateau = tf.keras.callbacks.ReduceLROnPlateau(monitor = 'val_loss', factor = 0.5, patience = 5, cooldown = 8, min_lr = 0.0001, verbose = 1)
mod_dcnn_no_aug_hist = mod_dcnn_no_aug.fit(ds_train,
batch_size = BATCH_SIZE,
epochs = n_epochs,
verbose = 0,
callbacks = [early_stop, reduce_lr_plateau],
validation_split = 0.0,
validation_data = ds_val,
shuffle = True,
class_weight = None,
sample_weight = None,
initial_epoch = 0,
steps_per_epoch = None,
validation_steps = None,
validation_batch_size = None,
validation_freq = val_freq)
y_pred_val_dcnn_no_aug_proba = mod_dcnn_no_aug.predict(ds_val, verbose = 0, callbacks = None)
y_pred_val_dcnn_no_aug = y_pred_val_dcnn_no_aug_proba.argmax(axis = 1)
y_pred_dcnn_no_aug_proba = mod_dcnn_no_aug.predict(np.stack(X_test.norm_flat.apply(flat_to_array)), verbose = 0, callbacks = None)
y_pred_dcnn_no_aug = y_pred_dcnn_no_aug_proba.argmax(axis = 1)
Epoch 12: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455. Epoch 24: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228. Epoch 36: ReduceLROnPlateau reducing learning rate to 0.0006249999860301614.
Now that the models have all been successfully trained and tuned, we can extract some fun and interesting information from different sections of the networks.
Note: Unless otherwise specified all the analysis will be from the model (named CNN in this project) from section 7.2.2.
By extracting the weights from the convolution layers, we can visualize the filter being applied to the input images. Here, this process is done to the first convolution layer, visualizing all 16 of the 3x3 kernels. For convenience, I plotted both the separated the RGB channels and the combined RGB filter.
# Extract weights from first convolution layer.
filters, biases = mod_cnn.layers[0].get_weights()
# Normalize values.
f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)
print('Filter Shape:', filters.shape)
# Plot all filters and channels.
n_filters = filters.shape[3]
fig, ax = plt.subplots(4, n_filters, sharey = True, sharex = True)
for i in range(n_filters):
filters_rgb = filters[:,:,:,i]
for j in range(3):
ax[j,i].imshow(filters_rgb[:,:,j], cmap = 'gray')
ax[j,i].set_xticks([])
ax[j,i].set_yticks([])
if i == 0:
ax[j,i].set_ylabel(j)
ax[3,i].imshow(filters_rgb)
ax[3,i].set_xticks([])
ax[3,i].set_yticks([])
ax[3,i].set_xlabel(i)
ax[3,0].set_ylabel('RGB')
fig.supylabel('Channel Index')
fig.supxlabel('Filter Index', y = 0.25)
fig.suptitle('First Convolution Layer Filters', y = 0.7)
fig.tight_layout(h_pad = -14, w_pad = 0.5)
plt.show()
Filter Shape: (3, 3, 3, 16)
Note, pixels that are weighted more show up as white and pixels that are not show up as black.
Very similar to sobel filters and the like, these filters have been trained and weighted to pick up simple features like horizontal, vertical lines, angled lines, etc but of course these are represented in RGB space so it is a little more difficult to visually identify their purpose.
For a more human-comprehendible visualization -- next, we can jump ahead in the network to visualize the output of the convolution blocks as a whole by accessing the output at the max pool layer that is placed at the end of each convolution block. By passing through an image into the network, instead of the individual kernels we just saw, these outputs equate to the down-sampled filter view of the image.
The code will output as follows:
# Indexes of maxpool layers at end of conv blocks.
pool_idx = [4, 11]
# Retrieve output of each layer.
outputs = [mod_cnn.layers[i].output for i in pool_idx]
mod_cnn_feat = tf.keras.Model(inputs = mod_cnn.inputs,
outputs = outputs)
# Load unseen image.
img = Image.open('./Data/Female/186.jpg')
img = img.resize((img_size, img_size), Image.Resampling.LANCZOS)
img = np.asarray(img)
img = np.expand_dims(img, axis = 0)
plt.imshow(img[0,:,:,:])
plt.show()
# Retrieve feature maps for image.
feature_maps = mod_cnn_feat.predict(img)
# Get number of filters from shape of filters = (16, 32).
n_filters = [feature_maps[0].shape[3], feature_maps[1].shape[3]]
# Iterate through each feature map.
for i, fmap in enumerate(feature_maps):
print(f'Convolution Block {i+1}')
print('Filter Shape:', fmap.shape)
fig, ax = plt.subplots(int(n_filters[i]/4), 4, figsize = ((4,4) if i == 0 else (4,8)))
# Iterate through number of filters in current fmap layer.
for j in range(n_filters[i]):
# Plot each filter feature map.
ax[j//4, j%4].set_xticks([])
ax[j//4, j%4].set_yticks([])
ax[j//4, j%4].imshow(fmap[0,:,:,j], cmap = 'gray')
plt.show()
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 36ms/step Convolution Block 1 Filter Shape: (1, 28, 28, 16)
Convolution Block 2 Filter Shape: (1, 14, 14, 32)
This is a great way to begin understanding what features the network has learned to weight heavily in its classification task.
In the first convolution block output we can see several filters that capture various structures:
The last plot showing the last convolution block shows what would be expected of an activation map view. We can see a much more contrasted and simplified view of this image. Some filters seem to be capture:
Another method to determine by how much a model weights certain features is Occlusion Sensitivity. Occlusion sensitivity, put simply, occludes or covers portions of an input image and passes it through the model, tracking the changes in class probabilities and thus building a heatmap of important areas of the image.
Two parameters are important in this process: The occluding patch size and the process by which the occlusion location is chosen.
You can implement this manually by randomly choosing occlusion locations, eventually processing each position, or even just iterate across the axis.
In this case, TensorFlow has an addon called tf-explain that makes the process easier and is implemented in a better way than the two simpler methods above.
The plots below occlusion sensitivity being performed on 4 unseen images and with 5 different occlusion patch sizes.
# List of unseen images to use. 2 Female, 2 Male.
unseen_eyes = ['./Data/Female/186.jpg',
'./Data/Female/2060.jpg',
'./Data/Male/186.jpg',
'./Data/Male/3154.jpg']
fig, ax = plt.subplots(6, 4, figsize = (9, 12))
# Iterate through the 4 random unseen eye images.
for i, file in enumerate(unseen_eyes):
# From TF-Explain: https://github.com/sicara/tf-explain/blob/master/examples/core/occlusion_sensitivity.py
img = tf.keras.preprocessing.image.load_img(file, target_size = (56, 56))
img = tf.keras.preprocessing.image.img_to_array(img)
explainer = OcclusionSensitivity()
# Set index for correct class.
class_index = 0 if i < 2 else 1
# Iterate different patch sizes and plot.
for j, patch in enumerate([2,4,8,16,24]):
# Compute Occlusion Sensitivity at different patch sizes.
explained = explainer.explain(([img], None), mod_cnn, class_index = class_index, patch_size = patch)
ax[j, i].set_title(f'{patch}')
ax[j, i].set_xticks([])
ax[j, i].set_yticks([])
ax[j, i].imshow(explained)
# Show original image at last row.
img = Image.open(file)
img = img.resize((img_size, img_size), Image.Resampling.LANCZOS)
ax[5, i].imshow(img)
ax[5, i].set_xticks([])
ax[5, i].set_yticks([])
ax[5, i].set_title(f'Class: {class_index}')
plt.show()
25/25 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step 7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step 2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 19ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step 25/25 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step 7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 15ms/step 2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step 25/25 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step 7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step 2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step 25/25 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step 7/7 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step 2/2 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 18ms/step
Female = 0, Male = 1
The two left columns are labeled as women and the two right columns are men.
Interestingly, and at least for this small sample, we can see some subtle difference in what features are being shown as important between the two classes. We can also see some changes in results as the occlusion patch size increase, thus showing results of the effects of larger features.
Female:
Male:
This plot brings up a lot of questions and insights about how this model is classifying images. It is quite possible that there are subtle differences in the dataset that are effecting it's "decisions". For instance, it might be useful to explore how balanced the hair color and makeup use is between the two classes. It could be possible that mascara use in the women is allowing the models to heavily favor using this feature for classifying the images. Futhermore, since the source of these images are from a IMDB-esque website most of these images are likely either professional headshots or movie posters. These facts are likely introducing a large amount of bias into this dataset.
Another valuable analysis tool is to extract the feature embeddings (layer outputs) and use Principal Component Analysis (PCA) to reduce the dimensions and plot the top two principal components. We can even plot each image on this PCA for an impressive visualization that might allow us to identify emerging complex patterns.
The images from the test set will be used for these plots and each image will be bordered with the color of the true class label (Female = Teal, Male = Purple). One version with the image and another as a scatter plot for easier visual parsing of the class distribution.
# Get feature vector from model.
def feature_extractor(model, layer_num):
# https://keras.io/getting_started/faq/#how-can-i-obtain-the-output-of-an-intermediate-layer-feature-extraction
extractor = tf.keras.Model(inputs = model.inputs,
outputs = [layer.output for layer in model.layers])
features_test = extractor(np.stack(X_test.norm_flat.apply(flat_to_array)))
feature_vec_cnn = []
# Take feature vector from layer_num.
for i in range(len(X_test)):
feature_vec_cnn.append(features_test[layer_num][i].numpy())
return feature_vec_cnn
def pca_plot(features, image_arrays, zoom = 0.5, cmap = None, photos = True): #zoom = 0.125
# Reduce dimensions to 2 using PCA.
pca = PCA(n_components = 2)
pca_fit = pca.fit_transform(features)
print(f'Explained Variance Ratios: {pca.explained_variance_ratio_}')
_, ax = plt.subplots(figsize = (20, 15), subplot_kw = {'aspect' : 'equal'})
ax.scatter(pca_fit[:, 0], pca_fit[:, 1], c = cmap, alpha = 0.8)
# Add eye photos to plot.
if photos == True:
for i, rgb_flat in enumerate(image_arrays):
# Load image.
image = Image.fromarray(rgb_flat.reshape(img_size,img_size,3))
# Zoom out.
im = OffsetImage(image, zoom = zoom)
# Set class label color for edge bbox.
bboxprops = ({'edgecolor' : cmap[i], 'lw' : 2} if cmap is not None else None)
anno_bbox = AnnotationBbox(offsetbox = im,
xy = pca_fit[i],
xycoords = 'data',
frameon = (bboxprops is not None),
pad = 0.075,
bboxprops = bboxprops)
ax.add_artist(anno_bbox)
ax.set_axis_off()
ax.axis('tight')
ax.set_title('Principal Component Analysis (PCA) of Eye Feature Embeddings')
return ax
# Teal = Women, Purple = Men
cmap_sex_all = ['#439A86' if sex == 0 else '#423e80' for sex in y_test]
First from the CNN (section 7.2.2.) we can extract and visualize the resulting embeddings coming out of the last convolution block and into the first dense layer.
Above each plot is the layer number and which model's embedding is being shown. Also, the resulting top 2 principal component's explained variance ratios.
feature_vec_cnn = feature_extractor(mod_cnn, 13)
# Plot 2nd to last fully connected layer.
print('CNN - Layer 13')
_ = pca_plot(feature_vec_cnn, X_test.rgb_flat.tolist(), cmap = cmap_sex_all)
plt.show()
_ = pca_plot(feature_vec_cnn, X_test.rgb_flat.tolist(), cmap = cmap_sex_all, photos = False)
plt.show()
CNN - Layer 13 Explained Variance Ratios: [0.20404717 0.09216744]
Explained Variance Ratios: [0.20404717 0.09216744]
Identifying combine characteristics from principal components is a highly imprecise art, but I believe we can see a few things along each axis here.
Now, we can do the same but on the last fully connected layer (dense) of the same model.
feature_vec_cnn = feature_extractor(mod_cnn, 16)
# Plot the last fully connected layer PCA.
print('CNN - Layer 16')
_ = pca_plot(feature_vec_cnn, X_test.rgb_flat.tolist(), cmap = cmap_sex_all)
plt.show()
_ = pca_plot(feature_vec_cnn, X_test.rgb_flat.tolist(), cmap = cmap_sex_all, photos = False)
plt.show()
CNN - Layer 16 Explained Variance Ratios: [0.54373284 0.03505089]
Explained Variance Ratios: [0.54373284 0.03505089]
The difference between this and the previous is drastic and the two clusters seem to have put more distance between them. Oddly we can see a new prevailing feature along the y-axis here.
Now to do the same thing but with the DCNN (section 7.2.3.). We will be extracting the output from the GlobalAveragePooling2D layer near the bottom of the network before prediction.
feature_vec_dcnn = feature_extractor(mod_dcnn, -2)
print('DCNN - Layer 123')
# Plot global pool layer.
_ = pca_plot(feature_vec_dcnn, X_test.rgb_flat.tolist(), cmap = cmap_sex_all)
plt.show()
_ = pca_plot(feature_vec_dcnn, X_test.rgb_flat.tolist(), cmap = cmap_sex_all, photos = False)
plt.show()
DCNN - Layer 123 Explained Variance Ratios: [0.77782455 0.10590087]
Explained Variance Ratios: [0.77782455 0.10590087]
The explained variance ratios a show an impressive ~87% between the two components and its easy to see that by how structured this PCA plot turned out. The easiest learners are at the edge of the long swooping shape.
It can be useful to examine the images that were misclassified and see if there are any obvious image differences or correlations between them that might give insights into why the model is misclassifying them.
# Get mask for all misclassified images in test set.
mask_missed_cnn = y_test != y_pred_cnn
# Sample 25 images from misclassified images.
missed_real_labels = y_test[mask_missed_cnn].sample(25, random_state = 11)
cmap_sex_missed = ['#439A86' if sex == 0 else '#423e80' for sex in missed_real_labels]
class_map = {0: 'Female', 1: 'Male'}
fig, ax = plt.subplots(5, 5, figsize = (10,10))
# Use index to retrieve images and loop through.
for i, img in enumerate(X_test.loc[missed_real_labels.index].rgb_flat):
ax[i//5, i%5].set_xticks([])
ax[i//5, i%5].set_yticks([])
ax[i//5, i%5].imshow(flat_to_array(img))
# Add border for real label.
border = plt.Rectangle((0, 0), 55, 55,
fill = False,
color = cmap_sex_missed[i],
linewidth = 3.5)
ax[i//5, i%5].add_patch(border)
# Add text for incorrect predictions.
ax[i//5, i%5].text(2, 53, 'Prediction:', fontsize = 9, color = '#e61405')
ax[i//5, i%5].text(53, 53, (f"{class_map[1 - missed_real_labels.iloc[i]]}"),
fontsize = 9,
weight = 'bold',
ha = 'right',
color = cmap_sex[class_map[1 - missed_real_labels.iloc[i]]]
).set_path_effects([path_effects.Stroke(linewidth=1, foreground='black'),
path_effects.Normal()])
plt.show()
Real Label: The colored border of the image indicates the real class, consistent with all other visualizations.
Predicted Label: The text printed on the image.
Looking through these sampled 25 (out of 29) of misclassified images, many of them share a few common characteristics.
Besides the obvious issues like the images with hair obscuring the eyes and images taken at odd angles, there likely aren't enough images in the training data that represent older individuals or people with glasses. There also seem to be quite a few examples of people with light colored or hard to see eyelashes. These hard learners (and others) might benefit from higher resolution images.
Now, it seems like a missed opportunity to not attempt to predict on some personal photos.
Below I will import a periocular image of myself and my wife. We both present as Male and Female respectively. Let's see if my model can correctly classify us as such.
# Import prediction images for fun.
predict_files = os.listdir('./Data/Predict/')
fig, ax = plt.subplots(1, len(predict_files))
for i, img in enumerate(predict_files):
img = Image.open(f'./Data/Predict/{img}')
img = img.resize((img_size, img_size), Image.Resampling.LANCZOS)
img = img.convert('RGB')
img = np.asarray(img)
img = np.expand_dims(img, axis = 0)
ax[i].imshow(img[0,:,:,:])
# Predict image class.
predict_proba = mod_cnn.predict(img, verbose = "auto", callbacks = None)
predict = predict_proba.argmax(axis = 1)
ax[i].set_xticks([])
ax[i].set_yticks([])
# Add border for real label.
border = plt.Rectangle((0, 0), 55, 55, fill = False, color = cmap_sex[class_map[predict[0]]], linewidth = 3.5)
ax[i].add_patch(border)
ax[i].text(2, 53, 'Prediction:', fontsize = 10, color = '#e61405')
ax[i].text(54, 53, (f"{class_map[predict[0]]}"), fontsize = 10, weight = 'bold', ha = 'right',
color = cmap_sex[class_map[predict[0]]]
).set_path_effects([path_effects.Stroke(linewidth=1, foreground='black'),
path_effects.Normal()])
plt.show()
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 14ms/step 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step
Well, no gender-bending surprises there, the model works great!
# Plot confusion matrices of all models.
def confusion_matrix_subplot(y_pred_dict):
model_class_vals = {}
fig, ax = plt.subplots(int(np.ceil(len(y_pred_dict) / 2)), 2, figsize = (7, 7), sharey = True)
for i, (model, y_pred) in enumerate(y_pred_dict.items()):
y_true = y_test
cm = confusion_matrix(y_true, y_pred)
disp = ConfusionMatrixDisplay(confusion_matrix = cm)
disp.plot(ax = ax[i//2, i%2])
disp.ax_.set_title(str(model))
disp.im_.colorbar.remove()
# Store classification metrics
# In the order [tn, fp, fn, tp]
model_class_vals[model] = list(confusion_matrix(y_true, y_pred, labels = y_true.unique().sort()).ravel())
# Check if odd number of models and delete last subplot if true.
if len(y_pred_dict) % 2 != 0:
fig.delaxes(ax[i//2, (len(y_pred_dict) % 3)])
fig.tight_layout()
plt.subplots_adjust(wspace = 0.2, hspace = 0)
fig.colorbar(disp.im_, ax = ax)
fig.suptitle('Confusion Matrices of All Models')
return model_class_vals
# Plot all model ROC curves.
def roc_curve_subplot(y_pred_proba_dict):
fig, ax = plt.subplots(1,1, figsize = (8,8))
for i, model in enumerate(y_pred_proba_dict.keys()):
y_pred_proba_temp = y_pred_proba_dict[model][:, 1]
fpr, tpr, _ = roc_curve(y_test, y_pred_proba_temp)
auc_score = auc(fpr, tpr)
ax.plot(fpr, tpr, label = f"AUC = {auc_score:0.3f} | {model}", alpha = 0.8)
# Add 45 degree line for 0.5 AUC.
ax.plot(np.arange(0, 1.1, 0.1), np.arange(0, 1.1, 0.1), 'k--')
ax.set_title('Test Set - ROC Curve')
ax.set_xlabel('False Positive Rate')
ax.set_ylabel('True Positive Rate')
plt.grid(alpha = 0.4)
plt.legend(loc = 'center right')
plt.show()
# Highlight the best model's test results green at each proportion.
def max_value_highlight(df):
max_test_rows = df.max()
is_max = (df == max_test_rows)
return ['background-color:green' if v else '' for v in is_max]
# Highlight the top two results in each column blue so that 2nd place is in blue after .apply().
def highlight_top_two(df):
# Sort values
sorted_df = df.sort_values(ascending = False)
top_two = sorted_df.iloc[:2]
# Mask
is_top_two = df.isin(top_two)
return ['background-color: blue' if v else '' for v in is_top_two]
For convenience, we can store the prediction arrays in dictionaries to be iterated over within the functions above.
y_pred_dict = {'KNN': y_pred_knn,
'FNN': y_pred_fnn,
'CNN': y_pred_cnn,
'DCNN': y_pred_dcnn}
y_pred_proba_dict = {'KNN': y_pred_knn_proba,
'FNN': y_pred_fnn_proba,
'CNN': y_pred_cnn_proba,
'DCNN': y_pred_dcnn_proba}
y_pred_val_dict = {'KNN': y_pred_val_knn,
'FNN': y_pred_val_fnn,
'CNN': y_pred_val_cnn,
'DCNN': y_pred_val_dcnn}
y_pred_no_aug_dict = {'KNN': y_pred_knn,
'FNN': y_pred_fnn_no_aug,
'CNN': y_pred_cnn_no_aug,
'DCNN': y_pred_dcnn_no_aug}
y_pred_val_no_aug_dict = {'KNN': y_pred_val_knn,
'FNN': y_pred_val_fnn_no_aug,
'CNN': y_pred_val_cnn_no_aug,
'DCNN': y_pred_val_dcnn_no_aug}
Here I can plot a confusion matrix for each model.
model_class_vals = confusion_matrix_subplot(y_pred_dict)
The performance of the CNN and DCNN are almost the same, though the CNN edges out the DCNN here just barely on the test set (which was the opposite on the validation set). It is also surprising to see that the two CNN-based models perform about equally on each class.
KNN:
FNN:
CNN:
DCNN:
roc_curve_subplot(y_pred_proba_dict)
The Area Under the Curve (AUC) scores of each of the models shows the same story here -- though the DCNN just edged out the CNN, but in reality the both performed about the same on the test set.
Now let's calculate the accuracy of each model and output the results into a dataframe. Also, we will evaluate each model on the non-augmented training dataset.
Accuracy was chosen because the dataset was class balanced and that was maintained through splitting into testing and validation sets as well.
# Non-augmented training images.
results_no_aug_dict = {}
for model, y_pred in y_pred_no_aug_dict.items():
val_acc = accuracy_score(y_val, y_pred_val_no_aug_dict[model])
test_acc = accuracy_score(y_test, y_pred)
results_no_aug_dict[model] = [val_acc, test_acc]
results_no_aug_df = pd.DataFrame().from_dict(results_no_aug_dict, orient = 'index', columns = ['Val Accuracy', 'Test Accuracy'])
results_no_aug_df = pd.concat([pd.DataFrame({'Val Accuracy': 0.5, 'Test Accuracy': 0.5}, index = ['Random Baseline']), results_no_aug_df])
results_no_aug_df.loc['KNN'] = [0, 0]
#results_no_aug_df.style.apply(highlight_top_two).apply(max_value_highlight)
# Augmented training images.
results_dict = {}
for model, y_pred in y_pred_dict.items():
val_acc = accuracy_score(y_val, y_pred_val_dict[model])
test_acc = accuracy_score(y_test, y_pred)
results_dict[model] = [val_acc, test_acc]
results_df = pd.DataFrame().from_dict(results_dict, orient = 'index', columns = ['Val Accuracy', 'Test Accuracy'])
results_df = pd.concat([pd.DataFrame({'Val Accuracy': 0.5, 'Test Accuracy': 0.5}, index = ['Random Baseline']), results_df])
#results_df.style.apply(highlight_top_two).apply(max_value_highlight)
results_concat = pd.concat([results_df, results_no_aug_df], axis = 1)
results_concat.columns = pd.MultiIndex.from_tuples(zip(['Augmented', 'Augmented.', 'Not Augmented', 'Not Augmented.'], results_concat.columns))
results_concat.style.apply(highlight_top_two).apply(max_value_highlight)
| Augmented | Augmented. | Not Augmented | Not Augmented. | |
|---|---|---|---|---|
| Val Accuracy | Test Accuracy | Val Accuracy | Test Accuracy | |
| Random Baseline | 0.500000 | 0.500000 | 0.500000 | 0.500000 |
| KNN | 0.773848 | 0.750804 | 0.000000 | 0.000000 |
| FNN | 0.843516 | 0.847267 | 0.863880 | 0.861736 |
| CNN | 0.944266 | 0.953376 | 0.931404 | 0.914791 |
| DCNN | 0.945338 | 0.950161 | 0.908896 | 0.885852 |
In each column, the best metric is highlighted in green and second best is blue. The two left columns are the models trained with the image augmentations and the two right columns are without augmentation.
First looking at the models trained using image augmentations:
Non-Augmented:
The results and analysis from this project demonstrate the effectiveness of deep convolutional neural networks (DCNNs) for classifying sex based on periocular images. An accuracy score of approximately 95% for sex identification using low-resolution 56x56 images shows promising results. The custom-built CNN performed the best on the test set with an accuracy score of ~95.34, while the DCNN modeled after ResNet-50 closely followed and outperformed on the validation set.
Some key takeaways from this project:
DCNN Efficacy: By training models of various architectures, it confirms that DCNNs can extract essential features from the periocular region to differentiate between sexes, even with shallower networks. The deep models displayed robust performance, especially after fine-tuning, and random image augmentation of the training data.
Feature Extraction Insights: In-depth analysis of model filters and feature maps revealed that certain filters in the early convolutional layers focused on basic visual features such as edges and textures, while deeper layers captured more complex patterns and structures relating to the eye and periocular region. The hierarchical layering of the network's architectures yield effective feature extraction and accurate sex classification. By visualizing convolutional layer filters, it was observed that the model learned distinct features which contributed to classification performance.
Data Augmentation's Impact: CNN-based models trained with image augmentation outperformed those without augmentation, showcasing improved generalization and reduced overfitting by adding size and diversity to the dataset. This was particularly evident because the dataset's size is relatively small for deep learning applications.
Not only were the models effective at sex classification using periocular images, but valuable insights were gained by exploring the feature extractions, revealing how networks process periocular images. The comparison of models with and without data augmentation also highlight the importance of using a sufficiently large and diverse dataset in deep learning. This project represents a strong first step in sex classification using eye images and informs the next steps for further understanding this task.
Resources that helped along the way in no particular order.
Alternative Datasets that would improve this project and allow for more nuanced segmentation of eye features:
Exported to HTML via command line using:
jupyter nbconvert Periocular_Sex_Classification.ipynb --to htmljupyter nbconvert Periocular_Sex_Classification.ipynb --to html --HTMLExporter.theme=dark